[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:42 dist-test-slave-q5qx krb5kdc[799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:42 dist-test-slave-q5qx krb5kdc[799](info): set up 2 sockets
Sep 29 18:35:42 dist-test-slave-q5qx krb5kdc[799](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 18:35:43.730471 792 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:35:43 dist-test-slave-q5qx krb5kdc[799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:35:43.853593 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:36863
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:36863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:43.960810 812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:43.961066 812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:43.961104 812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:43.964985 812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:43.965039 812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:43.965056 812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:43.965071 812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:43.965085 812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:43.970196 812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:36863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:43.971210 812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:43.972220 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:43.981091 818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:43.981073 817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:43.981150 820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:43.981263 812 server_base.cc:1034] running on GCE node
I20260929 18:35:43.982115 812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:43.983269 812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:43.984483 812 hybrid_clock.cc:648] HybridClock initialized: now 1790706943984427 us; error 68 us; skew 500 ppm
Sep 29 18:35:43 dist-test-slave-q5qx krb5kdc[799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:43.987401 812 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:35:43.988349 812 webserver.cc:460] Webserver started at http://127.0.198.62:43133/ using document root <none> and password file <none>
I20260929 18:35:43.988907 812 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:43.988976 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:43.989184 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:43.991317 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "b36d92dd45dd4f15a9480bc1e32cdde9"
format_stamp: "Formatted at 2026-09-29 18:35:43 on dist-test-slave-q5qx"
server_key: "955cf770365d44012c3e7d122f07960b"
server_key_iv: "92e7fa971d8a1fb4c33c5b17db25ff75"
server_key_version: "encryptionkey@0"
I20260929 18:35:43.991889 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "b36d92dd45dd4f15a9480bc1e32cdde9"
format_stamp: "Formatted at 2026-09-29 18:35:43 on dist-test-slave-q5qx"
server_key: "955cf770365d44012c3e7d122f07960b"
server_key_iv: "92e7fa971d8a1fb4c33c5b17db25ff75"
server_key_version: "encryptionkey@0"
I20260929 18:35:43.995836 812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:43.998521 827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:43.999997 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:44.000123 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "b36d92dd45dd4f15a9480bc1e32cdde9"
format_stamp: "Formatted at 2026-09-29 18:35:43 on dist-test-slave-q5qx"
server_key: "955cf770365d44012c3e7d122f07960b"
server_key_iv: "92e7fa971d8a1fb4c33c5b17db25ff75"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.000227 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:44.023622 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:44.026916 812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:44.027113 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:44.034097 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:36863
I20260929 18:35:44.034121 879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:36863 every 8 connection(s)
I20260929 18:35:44.035152 812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:44.038101 880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:44.040503 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 812
I20260929 18:35:44.040653 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:35:44.041927 792 external_mini_cluster.cc:1428] Setting key bf76dd5a1c776e2b06145738052dbc21
I20260929 18:35:44.044039 880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9: Bootstrap starting.
I20260929 18:35:44.047051 880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:44.047844 880 log.cc:824] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:44.049741 880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9: No bootstrap required, opened a new log
I20260929 18:35:44.053233 880 raft_consensus.cc:348] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 36863 } }
I20260929 18:35:44.053457 880 raft_consensus.cc:374] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:44.053514 880 raft_consensus.cc:729] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36d92dd45dd4f15a9480bc1e32cdde9, State: Initialized, Role: FOLLOWER
I20260929 18:35:44.054044 880 consensus_queue.cc:260] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 36863 } }
I20260929 18:35:44.054194 880 raft_consensus.cc:388] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:44.054265 880 raft_consensus.cc:482] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:44.054359 880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:44.055290 880 raft_consensus.cc:504] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 36863 } }
I20260929 18:35:44.055629 880 leader_election.cc:302] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b36d92dd45dd4f15a9480bc1e32cdde9; no voters:
I20260929 18:35:44.055938 880 leader_election.cc:288] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:44.056039 885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:44.056247 885 raft_consensus.cc:686] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [term 1 LEADER]: Becoming Leader. State: Replica: b36d92dd45dd4f15a9480bc1e32cdde9, State: Running, Role: LEADER
I20260929 18:35:44.056558 885 consensus_queue.cc:237] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 36863 } }
I20260929 18:35:44.057322 880 sys_catalog.cc:564] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:44.058102 886 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 36863 } } }
I20260929 18:35:44.058229 886 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:44.058560 887 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36d92dd45dd4f15a9480bc1e32cdde9. Latest consensus state: current_term: 1 leader_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36d92dd45dd4f15a9480bc1e32cdde9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 36863 } } }
I20260929 18:35:44.058646 887 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:44.059190 894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:44.062542 894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:44.067232 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.045238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.198.62:36863)
0929 18:35:44.045693 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:44.045712 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:44.046816 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 18:35:44.050091 (+ 3275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:44.050107 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:44.050442 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:44.050741 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:44.052873 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.053935 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.055225 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.055722 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.055860 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.061189 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:44.061196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:44.061213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:44.061248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:44.063310 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.063967 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.063974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.063980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.064133 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.064528 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.064531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.064532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.064936 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:44.065123 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:44.065698 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 18:35:44.066023 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":136,"threads_started":1}
I20260929 18:35:44.068814 894 catalog_manager.cc:1347] Generated new cluster ID: 7c65ab7bc7684db887501de87dc01d53
I20260929 18:35:44.068872 894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:44.086141 894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:44.087119 894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:44.094061 894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b36d92dd45dd4f15a9480bc1e32cdde9: Generated new TSK 0
I20260929 18:35:44.094827 894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:35:44.129400 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:36863
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:44.238637 906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:44.238920 906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:44.238965 906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:44.243072 906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:44.243148 906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:44.243209 906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:35:44.248876 906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:36863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:44.250041 906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:44.251034 906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:44.260529 912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:44.260489 914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:44.260653 906 server_base.cc:1034] running on GCE node
W20260929 18:35:44.260531 911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:44.261206 906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:44.261888 906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:44.263053 906 hybrid_clock.cc:648] HybridClock initialized: now 1790706944263036 us; error 33 us; skew 500 ppm
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:44.271903 906 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:35:44.272864 906 webserver.cc:460] Webserver started at http://127.0.198.1:43365/ using document root <none> and password file <none>
I20260929 18:35:44.273411 906 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:44.273469 906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:44.273631 906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:44.275750 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "ef4283fb5c034bd5abc08c4bd09651ed"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "61ee82735336eed092f6fd42da1540ef"
server_key_iv: "75e4b480d5ddff70a3e5af401cc6eeb7"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.276289 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "ef4283fb5c034bd5abc08c4bd09651ed"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "61ee82735336eed092f6fd42da1540ef"
server_key_iv: "75e4b480d5ddff70a3e5af401cc6eeb7"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.279975 906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:35:44.282390 921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.283912 906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:44.284016 906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "ef4283fb5c034bd5abc08c4bd09651ed"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "61ee82735336eed092f6fd42da1540ef"
server_key_iv: "75e4b480d5ddff70a3e5af401cc6eeb7"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.284109 906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:44.296782 906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:44.299908 906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:44.300063 906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:44.300712 906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:44.301522 906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:44.301568 906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.301606 906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:44.301626 906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.310444 906 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:43517
I20260929 18:35:44.310477 1034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:43517 every 8 connection(s)
I20260929 18:35:44.311365 906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:44.316008 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 906
I20260929 18:35:44.316094 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:44.316318 792 external_mini_cluster.cc:1428] Setting key 4bc4a859791cc4fab8dcd768f03f6ac5
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:35:44.325606 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.313074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:35423 (local address 127.0.198.62:36863)
0929 18:35:44.313198 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:44.313203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:44.314054 (+ 851us) server_negotiation.cc:407] Connection header received
0929 18:35:44.315397 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:44.315400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:44.315459 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:44.315529 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:44.316862 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.317507 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.318842 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.319094 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.319186 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.322011 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:44.322015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:44.322018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:44.322042 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:44.323609 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.324107 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.324110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.324111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.324183 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.324506 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.324509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.324511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.324686 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:44.324768 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:44.325321 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 18:35:44.325447 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:35:44.326581 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.313392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:36863 (local address 127.0.198.1:35423)
0929 18:35:44.313879 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:44.313927 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:44.315179 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:44.315720 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:44.315727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:44.316090 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:44.316671 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:44.316691 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.317654 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.317657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:44.318709 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:44.318720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.319241 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.319245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:44.319496 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.320153 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:44.320169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:44.321800 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:44.323732 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:44.323738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:44.323753 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:44.324009 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:44.324269 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:44.324272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:44.324274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:44.324420 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:44.324785 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:44.324791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:44.325076 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 18:35:44.325265 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:35:44.325608 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:35:44.327957 1035 heartbeater.cc:347] Connected to a master server at 127.0.198.62:36863
I20260929 18:35:44.328183 1035 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:44.328715 1035 heartbeater.cc:511] Master 127.0.198.62:36863 requested a full tablet report, sending...
I20260929 18:35:44.330312 844 ts_manager.cc:194] Registered new tserver with Master: ef4283fb5c034bd5abc08c4bd09651ed (127.0.198.1:43517)
I20260929 18:35:44.331455 844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:35423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:35:44.342651 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:36863
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:44.451088 1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:44.451354 1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:44.451396 1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:44.455338 1040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:44.455394 1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:44.455454 1040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:35:44.460884 1040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:36863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:44.461865 1040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:44.462860 1040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:44.472100 1046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:44.472306 1040 server_base.cc:1034] running on GCE node
W20260929 18:35:44.472136 1048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:44.472110 1045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:44.473012 1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:44.473912 1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:44.475131 1040 hybrid_clock.cc:648] HybridClock initialized: now 1790706944475100 us; error 43 us; skew 500 ppm
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:44.478364 1040 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:35:44.479242 1040 webserver.cc:460] Webserver started at http://127.0.198.2:33641/ using document root <none> and password file <none>
I20260929 18:35:44.479754 1040 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:44.479809 1040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:44.480005 1040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:44.482061 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "da53645d448f4b1a8697b1655b507a59"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "4601212d89414780298a3456e50dbf87"
server_key_iv: "6f6fe10f45b701e43db475209c3f4260"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.482604 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "da53645d448f4b1a8697b1655b507a59"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "4601212d89414780298a3456e50dbf87"
server_key_iv: "6f6fe10f45b701e43db475209c3f4260"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.486264 1040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:44.488694 1055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.489918 1040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:44.490088 1040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "da53645d448f4b1a8697b1655b507a59"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "4601212d89414780298a3456e50dbf87"
server_key_iv: "6f6fe10f45b701e43db475209c3f4260"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.490190 1040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:44.504186 1040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:44.507220 1040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:44.507396 1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:44.508033 1040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:44.508843 1040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:44.508889 1040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.508941 1040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:44.508968 1040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.517755 1040 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:35721
I20260929 18:35:44.517791 1168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:35721 every 8 connection(s)
I20260929 18:35:44.518697 1040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:44.528640 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1040
I20260929 18:35:44.528744 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:44.529026 792 external_mini_cluster.cc:1428] Setting key 6c2b0b07a36b6daa03a01e7ccf2795ad
I20260929 18:35:44.532004 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.520358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:44941 (local address 127.0.198.62:36863)
0929 18:35:44.520473 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:44.520476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:44.521394 (+ 918us) server_negotiation.cc:407] Connection header received
0929 18:35:44.522729 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:44.522732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:44.522780 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:44.522858 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:44.523962 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.524482 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.525632 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.525855 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.525941 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.528310 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:44.528313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:44.528315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:44.528350 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:44.529864 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.530464 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.530467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.530469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.530555 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.530869 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.530872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.530874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.531099 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:44.531191 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:44.531719 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 18:35:44.531831 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:35:44.533140 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.520764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:36863 (local address 127.0.198.2:44941)
0929 18:35:44.521236 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:44.521277 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:44.522512 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:44.523013 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:44.523019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:44.523353 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:44.523807 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:44.523819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.524599 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.524601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:44.525507 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:44.525516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.526030 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.526034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:44.526176 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.526739 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:44.526754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:44.528119 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:44.529998 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:44.530005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:44.530019 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:44.530370 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:44.530641 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:44.530643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:44.530644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:44.530775 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:44.531189 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:44.531194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:44.531426 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 18:35:44.531646 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 18:35:44.531952 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":140,"threads_started":1}
I20260929 18:35:44.534279 1169 heartbeater.cc:347] Connected to a master server at 127.0.198.62:36863
I20260929 18:35:44.534576 1169 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:44.535142 1169 heartbeater.cc:511] Master 127.0.198.62:36863 requested a full tablet report, sending...
I20260929 18:35:44.536240 844 ts_manager.cc:194] Registered new tserver with Master: da53645d448f4b1a8697b1655b507a59 (127.0.198.2:35721)
I20260929 18:35:44.536794 844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:44941
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:35:44.558161 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:36863
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:44.668305 1174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:44.668587 1174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:44.668649 1174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:44.672711 1174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:44.672775 1174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:44.672853 1174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:35:44.678123 1174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:36863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:44.679167 1174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:44.680162 1174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:44.689558 1179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:44.689575 1180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:44.689783 1182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:44.689800 1174 server_base.cc:1034] running on GCE node
I20260929 18:35:44.690466 1174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:44.691069 1174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:44.692224 1174 hybrid_clock.cc:648] HybridClock initialized: now 1790706944692187 us; error 52 us; skew 500 ppm
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:44.695595 1174 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:35:44.696458 1174 webserver.cc:460] Webserver started at http://127.0.198.3:35279/ using document root <none> and password file <none>
I20260929 18:35:44.696996 1174 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:44.697063 1174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:44.697240 1174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:44.699316 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "019e62f91caa472788fa6c63794fbca9"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "bc095833cc3cf19740ea8db12a2bb9ed"
server_key_iv: "daa7433304664b0480bd110eacbb4730"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.699843 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "019e62f91caa472788fa6c63794fbca9"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "bc095833cc3cf19740ea8db12a2bb9ed"
server_key_iv: "daa7433304664b0480bd110eacbb4730"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.703497 1174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:44.705766 1189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.707113 1174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:44.707211 1174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "019e62f91caa472788fa6c63794fbca9"
format_stamp: "Formatted at 2026-09-29 18:35:44 on dist-test-slave-q5qx"
server_key: "bc095833cc3cf19740ea8db12a2bb9ed"
server_key_iv: "daa7433304664b0480bd110eacbb4730"
server_key_version: "encryptionkey@0"
I20260929 18:35:44.707321 1174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:44.717571 1174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:44.720584 1174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:44.720752 1174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:44.721349 1174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:44.722213 1174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:44.722262 1174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.722301 1174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:44.722316 1174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:44.731334 1174 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:44887
I20260929 18:35:44.731433 1302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:44887 every 8 connection(s)
I20260929 18:35:44.732342 1174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:44.734547 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1174
I20260929 18:35:44.734633 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:44.734858 792 external_mini_cluster.cc:1428] Setting key 96237219e616dbbd6ac0a79b000193c7
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:44.746117 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.734110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:57855 (local address 127.0.198.62:36863)
0929 18:35:44.734250 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:44.734255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:44.735217 (+ 962us) server_negotiation.cc:407] Connection header received
0929 18:35:44.736772 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:44.736775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:44.736824 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:44.736905 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:44.738215 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.738767 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.739882 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.740095 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.740172 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.742610 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:44.742613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:44.742615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:44.742636 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:44.743967 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.744545 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.744548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.744550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.744636 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.745021 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.745024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.745025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.745197 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:44.745270 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:44.745856 (+ 586us) server_negotiation.cc:299] Negotiation successful
0929 18:35:44.745951 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:35:44.746942 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.734541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:36863 (local address 127.0.198.3:57855)
0929 18:35:44.735047 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:44.735101 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:44.736581 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:44.737104 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:44.737111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:44.737583 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:44.738093 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:44.738108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.738902 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.738905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:44.739770 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:44.739779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.740218 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.740221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:44.740428 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.740938 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:44.740954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:44.742441 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:44.744069 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:44.744074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:44.744086 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:44.744328 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:44.744745 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:44.744748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:44.744749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:44.744931 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:44.745304 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:44.745309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:44.745614 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 18:35:44.745771 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 18:35:44.746081 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
I20260929 18:35:44.748070 1303 heartbeater.cc:347] Connected to a master server at 127.0.198.62:36863
I20260929 18:35:44.748303 1303 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:44.748791 1303 heartbeater.cc:511] Master 127.0.198.62:36863 requested a full tablet report, sending...
I20260929 18:35:44.749792 844 ts_manager.cc:194] Registered new tserver with Master: 019e62f91caa472788fa6c63794fbca9 (127.0.198.3:44887)
I20260929 18:35:44.750357 844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:57855
I20260929 18:35:44.760874 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:35:44.786685 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.776280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.0.198.1:43517)
0929 18:35:44.777097 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:44.777107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:44.777130 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:35:44.777179 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:44.777184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:44.777454 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:44.777579 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:44.778292 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.779201 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.780106 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.780539 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.780624 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.783043 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:44.783052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:44.783066 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:44.783090 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:44.784665 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.785162 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.785168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.785169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.785259 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.785566 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.785568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.785570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.785816 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:44.786031 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:44.786297 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 18:35:44.786534 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":627,"threads_started":1}
W20260929 18:35:44.787689 949 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59994
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:35:44.812565 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:44.804452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.0.198.1:43517)
0929 18:35:44.804611 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:44.804615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:44.804695 (+ 80us) server_negotiation.cc:407] Connection header received
0929 18:35:44.804792 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:44.804795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:44.804872 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:44.804959 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:44.805686 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.806322 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.807137 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:44.807362 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:44.807450 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:44.809606 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:44.809610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:44.809612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:44.809636 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:44.811000 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.811409 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.811412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.811413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.811563 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:44.811801 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:44.811804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:44.811806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:44.812000 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:44.812110 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:44.812323 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:35:44.812436 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:35:44.813651 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 906
I20260929 18:35:44.820617 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1040
I20260929 18:35:44.825948 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1174
I20260929 18:35:44.830556 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 812
2026-09-29T18:35:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2076 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[1317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[1317](info): set up 2 sockets
Sep 29 18:35:44 dist-test-slave-q5qx krb5kdc[1317](info): commencing operation
krb5kdc: starting...
W20260929 18:35:45.781471 792 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:35:45 dist-test-slave-q5qx krb5kdc[1317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:35:45.897873 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41127
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:46.005811 1330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:46.006146 1330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:46.006210 1330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:46.010243 1330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:46.010317 1330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:46.010344 1330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:46.010365 1330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:46.010483 1330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:46.016675 1330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:46.017791 1330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:46.018867 1330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:46.027797 1335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:46.027786 1336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:46.028015 1330 server_base.cc:1034] running on GCE node
W20260929 18:35:46.027786 1338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:46.028827 1330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:46.030114 1330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:46.031291 1330 hybrid_clock.cc:648] HybridClock initialized: now 1790706946031266 us; error 42 us; skew 500 ppm
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:46.034216 1330 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:35:46.035167 1330 webserver.cc:460] Webserver started at http://127.0.198.62:44327/ using document root <none> and password file <none>
I20260929 18:35:46.035734 1330 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:46.035797 1330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:46.035990 1330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:46.038133 1330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "658cba950fcf47059e7f47bc158eb940"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "aa478cb4a029457c5d045682859ed6ce"
server_key_iv: "61dca40df1007eb196bba36fd6810040"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.038686 1330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "658cba950fcf47059e7f47bc158eb940"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "aa478cb4a029457c5d045682859ed6ce"
server_key_iv: "61dca40df1007eb196bba36fd6810040"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.042606 1330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:35:46.045125 1345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.046523 1330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:46.046636 1330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "658cba950fcf47059e7f47bc158eb940"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "aa478cb4a029457c5d045682859ed6ce"
server_key_iv: "61dca40df1007eb196bba36fd6810040"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.046749 1330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:46.074496 1330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:46.077809 1330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:46.078008 1330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:46.084926 1330 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:41127
I20260929 18:35:46.084992 1397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:41127 every 8 connection(s)
I20260929 18:35:46.085882 1330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:46.088749 1398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:46.094193 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1330
I20260929 18:35:46.094303 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:35:46.094379 1398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940: Bootstrap starting.
I20260929 18:35:46.094586 792 external_mini_cluster.cc:1428] Setting key 806da69e8a036f56772e7ca8afb4fce4
I20260929 18:35:46.097010 1398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:46.097891 1398 log.cc:824] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:46.100214 1398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940: No bootstrap required, opened a new log
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:46.103943 1398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658cba950fcf47059e7f47bc158eb940" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41127 } }
I20260929 18:35:46.104189 1398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:46.104243 1398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658cba950fcf47059e7f47bc158eb940, State: Initialized, Role: FOLLOWER
I20260929 18:35:46.104733 1398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658cba950fcf47059e7f47bc158eb940" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41127 } }
I20260929 18:35:46.104902 1398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:46.104974 1398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:46.105058 1398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:46.106153 1398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658cba950fcf47059e7f47bc158eb940" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41127 } }
I20260929 18:35:46.106534 1398 leader_election.cc:302] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 658cba950fcf47059e7f47bc158eb940; no voters:
I20260929 18:35:46.106873 1398 leader_election.cc:288] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:46.106966 1403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:46.107187 1403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [term 1 LEADER]: Becoming Leader. State: Replica: 658cba950fcf47059e7f47bc158eb940, State: Running, Role: LEADER
I20260929 18:35:46.107560 1403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658cba950fcf47059e7f47bc158eb940" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41127 } }
I20260929 18:35:46.108202 1398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:46.109332 1405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658cba950fcf47059e7f47bc158eb940. Latest consensus state: current_term: 1 leader_uuid: "658cba950fcf47059e7f47bc158eb940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658cba950fcf47059e7f47bc158eb940" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41127 } } }
I20260929 18:35:46.109549 1405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:46.109946 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.095943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.0.198.62:41127)
0929 18:35:46.096398 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.096422 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.096463 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:35:46.097327 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.097352 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.097774 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.098160 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.098920 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.100352 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.101272 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.102019 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.102173 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.104630 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.104636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.104653 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.104685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.106668 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.107224 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.107230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.107236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.107386 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.107741 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.107744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.107745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.108171 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.108347 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.108628 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.108927 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":122,"threads_started":1}
I20260929 18:35:46.110024 1412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:46.111002 1404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "658cba950fcf47059e7f47bc158eb940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658cba950fcf47059e7f47bc158eb940" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41127 } } }
I20260929 18:35:46.111244 1404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:46.112900 1412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:46.118180 1412 catalog_manager.cc:1347] Generated new cluster ID: 466efdd4bf4b4ad5a805e8dda776698c
I20260929 18:35:46.118264 1412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:46.130306 1412 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:46.131599 1412 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:46.137516 1412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 658cba950fcf47059e7f47bc158eb940: Generated new TSK 0
I20260929 18:35:46.138473 1412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:35:46.169370 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41127
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:46.279443 1424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:46.279736 1424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:46.279781 1424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:46.283860 1424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:46.283933 1424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:46.283995 1424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:35:46.289390 1424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:46.290488 1424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:46.291607 1424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:46.301119 1430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:46.301103 1429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:46.301218 1432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:46.301261 1424 server_base.cc:1034] running on GCE node
I20260929 18:35:46.301818 1424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:46.302469 1424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:46.303630 1424 hybrid_clock.cc:648] HybridClock initialized: now 1790706946303610 us; error 35 us; skew 500 ppm
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:46.306221 1424 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:35:46.307214 1424 webserver.cc:460] Webserver started at http://127.0.198.1:34499/ using document root <none> and password file <none>
I20260929 18:35:46.307734 1424 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:46.307796 1424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:46.307984 1424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:46.310107 1424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "0ce20c82ff8e4ca8b03713e461bd03dd"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "417c63ff0a6cd1550ac9aa47e45118bd"
server_key_iv: "e79d37f53df160247ffee0f561838d54"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.310623 1424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "0ce20c82ff8e4ca8b03713e461bd03dd"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "417c63ff0a6cd1550ac9aa47e45118bd"
server_key_iv: "e79d37f53df160247ffee0f561838d54"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.314205 1424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:46.316679 1439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.317920 1424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:46.318068 1424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "0ce20c82ff8e4ca8b03713e461bd03dd"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "417c63ff0a6cd1550ac9aa47e45118bd"
server_key_iv: "e79d37f53df160247ffee0f561838d54"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.318171 1424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:46.342969 1424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:46.345903 1424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:46.346093 1424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:46.346669 1424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:46.347560 1424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:46.347616 1424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.347675 1424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:46.347699 1424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.356041 1424 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:36495
I20260929 18:35:46.356067 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:36495 every 8 connection(s)
I20260929 18:35:46.356899 1424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:46.365574 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1424
I20260929 18:35:46.365682 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:46.365909 792 external_mini_cluster.cc:1428] Setting key 6b5649d52046fb7f20e3806dce7b3297
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:46.370225 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.358801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:53695 (local address 127.0.198.62:41127)
0929 18:35:46.358983 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.358987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.359758 (+ 771us) server_negotiation.cc:407] Connection header received
0929 18:35:46.361097 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.361100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.361154 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.361226 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.362418 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.362939 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.364052 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.364281 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.364380 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.366836 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.366840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.366843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.366867 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.368288 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.368774 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.368777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.368779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.368851 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.369166 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.369170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.369171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.369344 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.369437 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.369921 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.370071 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:35:46.371039 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.359143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41127 (local address 127.0.198.1:53695)
0929 18:35:46.359602 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.359645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.360892 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.361399 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.361406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.361750 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:46.362281 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.362295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.363051 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.363054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.363939 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.363948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.364420 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.364424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.364584 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.365122 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:46.365138 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:46.366684 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:46.368399 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.368405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.368422 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.368684 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.368928 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.368931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.368933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.369080 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.369458 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:46.369465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:46.369707 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.369874 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.370233 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260929 18:35:46.372315 1553 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41127
I20260929 18:35:46.372540 1553 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:46.373004 1553 heartbeater.cc:511] Master 127.0.198.62:41127 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:35:46.374735 1362 ts_manager.cc:194] Registered new tserver with Master: 0ce20c82ff8e4ca8b03713e461bd03dd (127.0.198.1:36495)
I20260929 18:35:46.375841 1362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:53695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:35:46.392992 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41127
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:46.502359 1558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:46.502666 1558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:46.502722 1558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:46.506774 1558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:46.506836 1558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:46.506902 1558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:35:46.512408 1558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:46.513456 1558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:46.514454 1558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:35:46.524475 1558 server_base.cc:1034] running on GCE node
W20260929 18:35:46.524425 1564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:46.524447 1563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:46.524425 1566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:46.525065 1558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:46.525745 1558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:46.526921 1558 hybrid_clock.cc:648] HybridClock initialized: now 1790706946526898 us; error 34 us; skew 500 ppm
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:46.529768 1558 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:35:46.530679 1558 webserver.cc:460] Webserver started at http://127.0.198.2:35473/ using document root <none> and password file <none>
I20260929 18:35:46.531206 1558 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:46.531265 1558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:46.531437 1558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:46.533510 1558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "e8bb45f1cd02438f8b5fc254bb861169"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "e0a190fa6dae4b77d37a128b2f2a7ac8"
server_key_iv: "e16bc22dabb481c6d20e4005419df9c2"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.534094 1558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "e8bb45f1cd02438f8b5fc254bb861169"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "e0a190fa6dae4b77d37a128b2f2a7ac8"
server_key_iv: "e16bc22dabb481c6d20e4005419df9c2"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.537935 1558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 18:35:46.540483 1573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.541839 1558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:35:46.541946 1558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "e8bb45f1cd02438f8b5fc254bb861169"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "e0a190fa6dae4b77d37a128b2f2a7ac8"
server_key_iv: "e16bc22dabb481c6d20e4005419df9c2"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.542114 1558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:46.553283 1558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:46.556749 1558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:46.556946 1558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:46.557518 1558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:46.558374 1558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:46.558421 1558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.558473 1558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:46.558501 1558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.567577 1558 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:44437
I20260929 18:35:46.567669 1686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:44437 every 8 connection(s)
I20260929 18:35:46.568507 1558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:46.569026 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1558
I20260929 18:35:46.569144 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:46.569438 792 external_mini_cluster.cc:1428] Setting key ca8bbad04784615df95038a1050050e2
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:35:46.582877 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.570556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:57229 (local address 127.0.198.62:41127)
0929 18:35:46.570697 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.570702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.571699 (+ 997us) server_negotiation.cc:407] Connection header received
0929 18:35:46.573092 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.573096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.573150 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.573221 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.574433 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.574937 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.576081 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.576380 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.576497 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.579045 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.579049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.579052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.579080 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.580468 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.581043 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.581045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.581047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.581125 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.581642 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.581645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.581646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.581841 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.581941 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.582638 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.582750 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:46.584048 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.570939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41127 (local address 127.0.198.2:57229)
0929 18:35:46.571537 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.571582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.572885 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.573404 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.573412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.573767 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:46.574296 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.574310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.575079 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.575083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.575967 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.575977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.576575 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.576579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.576744 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.577321 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:46.577337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:46.578842 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:46.580593 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.580599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.580614 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.580904 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.581234 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.581237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.581239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.581520 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.581996 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:46.582003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:46.582320 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.582523 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.582901 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":142,"threads_started":1}
I20260929 18:35:46.585247 1687 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41127
I20260929 18:35:46.585479 1687 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:46.586091 1687 heartbeater.cc:511] Master 127.0.198.62:41127 requested a full tablet report, sending...
I20260929 18:35:46.587128 1362 ts_manager.cc:194] Registered new tserver with Master: e8bb45f1cd02438f8b5fc254bb861169 (127.0.198.2:44437)
I20260929 18:35:46.587684 1362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:57229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:35:46.597520 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41127
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:46.706640 1692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:46.706905 1692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:46.706967 1692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:46.711099 1692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:46.711166 1692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:46.711246 1692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:35:46.716576 1692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:46.717676 1692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:46.718808 1692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:46.728366 1700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:46.728360 1698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:46.728435 1692 server_base.cc:1034] running on GCE node
W20260929 18:35:46.728366 1697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:46.729022 1692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:46.729635 1692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:46.730813 1692 hybrid_clock.cc:648] HybridClock initialized: now 1790706946730783 us; error 47 us; skew 500 ppm
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:46.733613 1692 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:35:46.734547 1692 webserver.cc:460] Webserver started at http://127.0.198.3:37997/ using document root <none> and password file <none>
I20260929 18:35:46.735090 1692 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:46.735153 1692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:46.735337 1692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:46.737390 1692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "691302f4da3b41f8ac5c4bfbea14177f"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "cb565b79797aef49b63684989572a91a"
server_key_iv: "8a1112d5f9b3881bec9ea9c67854180c"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.737928 1692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "691302f4da3b41f8ac5c4bfbea14177f"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "cb565b79797aef49b63684989572a91a"
server_key_iv: "8a1112d5f9b3881bec9ea9c67854180c"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.741726 1692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:35:46.744534 1707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.745842 1692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:46.746002 1692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "691302f4da3b41f8ac5c4bfbea14177f"
format_stamp: "Formatted at 2026-09-29 18:35:46 on dist-test-slave-q5qx"
server_key: "cb565b79797aef49b63684989572a91a"
server_key_iv: "8a1112d5f9b3881bec9ea9c67854180c"
server_key_version: "encryptionkey@0"
I20260929 18:35:46.746126 1692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:46.763684 1692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:46.766904 1692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:46.767092 1692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:46.767663 1692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:46.768466 1692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:46.768512 1692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.768561 1692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:46.768590 1692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.777187 1692 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:44697
I20260929 18:35:46.777249 1820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:44697 every 8 connection(s)
I20260929 18:35:46.778131 1692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:46.783478 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1692
I20260929 18:35:46.783607 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:46.783833 792 external_mini_cluster.cc:1428] Setting key e17c71535350c5639c1caeb2bf588330
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:46.791543 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.780034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:47621 (local address 127.0.198.62:41127)
0929 18:35:46.780205 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.780210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.781080 (+ 870us) server_negotiation.cc:407] Connection header received
0929 18:35:46.782356 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.782359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.782408 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.782492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.783684 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.784271 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.785413 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.785632 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.785720 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.788191 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.788194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.788197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.788221 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.789559 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.790080 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.790084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.790086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.790166 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.790484 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.790487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.790489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.790671 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.790752 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.791253 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.791383 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:35:46.792339 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.780461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41127 (local address 127.0.198.3:47621)
0929 18:35:46.780905 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.780950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.782157 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.782658 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.782664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.783035 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:46.783526 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.783540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.784392 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.784395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.785293 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.785303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.785764 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.785767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.785928 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.786553 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:46.786569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:46.788029 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:46.789681 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.789687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.789705 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.789995 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.790247 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.790250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.790252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.790391 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.790764 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:46.790771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:46.791024 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.791184 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.791514 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260929 18:35:46.793573 1821 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41127
I20260929 18:35:46.793824 1821 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:46.794353 1821 heartbeater.cc:511] Master 127.0.198.62:41127 requested a full tablet report, sending...
I20260929 18:35:46.795396 1362 ts_manager.cc:194] Registered new tserver with Master: 691302f4da3b41f8ac5c4bfbea14177f (127.0.198.3:44697)
I20260929 18:35:46.795941 1362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:47621
I20260929 18:35:46.797547 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:46.807937 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.800876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.0.198.62:41127)
0929 18:35:46.801008 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.801013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.801190 (+ 177us) server_negotiation.cc:407] Connection header received
0929 18:35:46.801303 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.801306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.801348 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.801409 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.802120 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.802647 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.803475 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.803687 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.803772 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.805035 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.805038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.805040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.805063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.806423 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.806829 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.806832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.806833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.806909 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.807132 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.807135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.807136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.807336 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.807432 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.807668 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.807772 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:46.812836 1362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:46.815923 1362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:46.829437 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.824138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.0.198.2:44437)
0929 18:35:46.824904 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.824915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.824940 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:35:46.825018 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.825023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.825246 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.825442 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.825754 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.826769 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.827946 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.828745 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.828842 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.828980 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.829216 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.829302 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":97,"threads_started":1}
I20260929 18:35:46.829445 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.823966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44437 (local address 127.0.0.1:33338)
0929 18:35:46.824594 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.824645 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.824768 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.825447 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.825453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.825478 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.825623 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.825630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.826883 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.826886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.827824 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.827832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.828839 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.828843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.828988 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.829033 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.829144 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.829262 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":24,"spinlock_wait_cycles":4992,"thread_start_us":93,"threads_started":1}
I20260929 18:35:46.831341 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.824216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:44697 (local address 127.0.0.1:37434)
0929 18:35:46.825415 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.825429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.825509 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.826154 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.826157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.826172 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.826268 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.826274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.827693 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.827696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.828770 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.828781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.830911 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.830915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.831011 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.831032 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.831128 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.831202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":88,"threads_started":1}
I20260929 18:35:46.831393 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.824379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.0.198.3:44697)
0929 18:35:46.824782 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.824794 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.825591 (+ 797us) server_negotiation.cc:407] Connection header received
0929 18:35:46.825668 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.825672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.825852 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.826014 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.826383 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.827562 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.830106 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.830775 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.830888 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.831025 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.831164 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.831251 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":220,"threads_started":1}
I20260929 18:35:46.831957 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.825051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.0.198.1:36495)
0929 18:35:46.825312 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.825326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.825858 (+ 532us) server_negotiation.cc:407] Connection header received
0929 18:35:46.825931 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.825936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.826138 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.826278 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.826474 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.827562 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.830075 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.830874 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.830972 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.831108 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.831762 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.831839 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260929 18:35:46.832436 1834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.824700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36495 (local address 127.0.0.1:47168)
0929 18:35:46.825689 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.825701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.825777 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.826273 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.826276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.826294 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.826374 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.826379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.828921 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.828923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.829947 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.829956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.831163 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.831166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.831241 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.831262 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.832218 (+ 956us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.832301 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260929 18:35:46.832855 1618 tablet_service.cc:1467] Processing CreateTablet for tablet fc468d955e87421180f26c977695bb85 (DEFAULT_TABLE table=test-table [id=a919088c3c31401e8986d25b6a5e2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:46.834087 1618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc468d955e87421180f26c977695bb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:46.834216 1755 tablet_service.cc:1467] Processing CreateTablet for tablet fc468d955e87421180f26c977695bb85 (DEFAULT_TABLE table=test-table [id=a919088c3c31401e8986d25b6a5e2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:46.835263 1755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc468d955e87421180f26c977695bb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:46.840988 1487 tablet_service.cc:1467] Processing CreateTablet for tablet fc468d955e87421180f26c977695bb85 (DEFAULT_TABLE table=test-table [id=a919088c3c31401e8986d25b6a5e2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:46.841949 1838 tablet_bootstrap.cc:492] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169: Bootstrap starting.
I20260929 18:35:46.842183 1487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc468d955e87421180f26c977695bb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:46.843571 1839 tablet_bootstrap.cc:492] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f: Bootstrap starting.
I20260929 18:35:46.844890 1838 tablet_bootstrap.cc:654] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:46.845724 1839 tablet_bootstrap.cc:654] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:46.845791 1838 log.cc:824] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:46.847038 1839 log.cc:824] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:46.847950 1838 tablet_bootstrap.cc:492] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169: No bootstrap required, opened a new log
I20260929 18:35:46.848137 1838 ts_tablet_manager.cc:1397] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260929 18:35:46.848666 1842 tablet_bootstrap.cc:492] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd: Bootstrap starting.
I20260929 18:35:46.848997 1839 tablet_bootstrap.cc:492] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f: No bootstrap required, opened a new log
I20260929 18:35:46.849179 1839 ts_tablet_manager.cc:1397] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:35:46.851365 1842 tablet_bootstrap.cc:654] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:46.851644 1838 raft_consensus.cc:348] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.851898 1838 raft_consensus.cc:374] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:46.851958 1838 raft_consensus.cc:729] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bb45f1cd02438f8b5fc254bb861169, State: Initialized, Role: FOLLOWER
I20260929 18:35:46.852375 1838 consensus_queue.cc:260] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.852995 1687 heartbeater.cc:503] Master 127.0.198.62:41127 was elected leader, sending a full tablet report...
I20260929 18:35:46.853013 1839 raft_consensus.cc:348] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.853273 1839 raft_consensus.cc:374] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:46.853322 1839 raft_consensus.cc:729] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691302f4da3b41f8ac5c4bfbea14177f, State: Initialized, Role: FOLLOWER
I20260929 18:35:46.853371 1838 ts_tablet_manager.cc:1428] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 18:35:46.853506 1842 log.cc:824] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:46.853746 1839 consensus_queue.cc:260] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.854686 1839 ts_tablet_manager.cc:1428] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 18:35:46.855027 1821 heartbeater.cc:503] Master 127.0.198.62:41127 was elected leader, sending a full tablet report...
I20260929 18:35:46.855690 1842 tablet_bootstrap.cc:492] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd: No bootstrap required, opened a new log
I20260929 18:35:46.855885 1842 ts_tablet_manager.cc:1397] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:35:46.859122 1842 raft_consensus.cc:348] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.859350 1842 raft_consensus.cc:374] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:46.859385 1842 raft_consensus.cc:729] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce20c82ff8e4ca8b03713e461bd03dd, State: Initialized, Role: FOLLOWER
I20260929 18:35:46.859756 1842 consensus_queue.cc:260] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.860374 1553 heartbeater.cc:503] Master 127.0.198.62:41127 was elected leader, sending a full tablet report...
I20260929 18:35:46.860599 1842 ts_tablet_manager.cc:1428] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:35:46.878599 1845 raft_consensus.cc:482] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:46.878823 1845 raft_consensus.cc:504] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.880386 1845 leader_election.cc:288] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8bb45f1cd02438f8b5fc254bb861169 (127.0.198.2:44437), 0ce20c82ff8e4ca8b03713e461bd03dd (127.0.198.1:36495)
I20260929 18:35:46.884881 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.880358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:37857 (local address 127.0.198.1:36495)
0929 18:35:46.880650 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.880655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.880667 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:35:46.880717 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.880720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.880768 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.880838 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.881216 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.881945 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.883604 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.884238 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.884332 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.884399 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.884712 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.884782 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260929 18:35:46.884874 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.880446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:35299 (local address 127.0.198.2:44437)
0929 18:35:46.880575 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.880579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.881146 (+ 567us) server_negotiation.cc:407] Connection header received
0929 18:35:46.881231 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.881233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.881283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.881353 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.881584 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.882217 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.883609 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.884217 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.884344 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.884399 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.884670 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.884752 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:46.884877 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.880256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36495 (local address 127.0.198.3:37857)
0929 18:35:46.880487 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.880503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.880596 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.880953 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.880956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.880974 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.881085 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.881092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.882076 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.882079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.883454 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.883465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.884362 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.884366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.884602 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.884621 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.884683 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.884733 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":53}
I20260929 18:35:46.884891 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.880256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44437 (local address 127.0.198.3:35299)
0929 18:35:46.881044 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.881057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.881145 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.881366 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.881368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.881381 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.881482 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.881487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.882311 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.882314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.883471 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.883477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.884362 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.884366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.884523 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.884552 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.884641 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.884715 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":372,"thread_start_us":141,"threads_started":1}
I20260929 18:35:46.885533 1641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468d955e87421180f26c977695bb85" candidate_uuid: "691302f4da3b41f8ac5c4bfbea14177f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bb45f1cd02438f8b5fc254bb861169" is_pre_election: true
I20260929 18:35:46.885536 1507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468d955e87421180f26c977695bb85" candidate_uuid: "691302f4da3b41f8ac5c4bfbea14177f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" is_pre_election: true
I20260929 18:35:46.885820 1641 raft_consensus.cc:2454] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 691302f4da3b41f8ac5c4bfbea14177f in term 0.
I20260929 18:35:46.885845 1507 raft_consensus.cc:2454] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 691302f4da3b41f8ac5c4bfbea14177f in term 0.
I20260929 18:35:46.886394 1708 leader_election.cc:302] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ce20c82ff8e4ca8b03713e461bd03dd, 691302f4da3b41f8ac5c4bfbea14177f, e8bb45f1cd02438f8b5fc254bb861169; no voters:
I20260929 18:35:46.886667 1845 raft_consensus.cc:2781] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:46.886751 1845 raft_consensus.cc:482] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:46.886790 1845 raft_consensus.cc:3037] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:46.887825 1845 raft_consensus.cc:504] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.888163 1845 leader_election.cc:288] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [CANDIDATE]: Term 1 election: Requested vote from peers e8bb45f1cd02438f8b5fc254bb861169 (127.0.198.2:44437), 0ce20c82ff8e4ca8b03713e461bd03dd (127.0.198.1:36495)
I20260929 18:35:46.888545 1641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468d955e87421180f26c977695bb85" candidate_uuid: "691302f4da3b41f8ac5c4bfbea14177f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bb45f1cd02438f8b5fc254bb861169"
I20260929 18:35:46.888682 1641 raft_consensus.cc:3037] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:46.889752 1641 raft_consensus.cc:2454] T fc468d955e87421180f26c977695bb85 P e8bb45f1cd02438f8b5fc254bb861169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691302f4da3b41f8ac5c4bfbea14177f in term 1.
I20260929 18:35:46.890066 1507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468d955e87421180f26c977695bb85" candidate_uuid: "691302f4da3b41f8ac5c4bfbea14177f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd"
I20260929 18:35:46.890113 1708 leader_election.cc:302] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 691302f4da3b41f8ac5c4bfbea14177f, e8bb45f1cd02438f8b5fc254bb861169; no voters:
I20260929 18:35:46.890192 1507 raft_consensus.cc:3037] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:46.890276 1845 raft_consensus.cc:2781] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:46.890524 1845 raft_consensus.cc:686] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [term 1 LEADER]: Becoming Leader. State: Replica: 691302f4da3b41f8ac5c4bfbea14177f, State: Running, Role: LEADER
I20260929 18:35:46.890892 1845 consensus_queue.cc:237] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } }
I20260929 18:35:46.891414 1507 raft_consensus.cc:2454] T fc468d955e87421180f26c977695bb85 P 0ce20c82ff8e4ca8b03713e461bd03dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691302f4da3b41f8ac5c4bfbea14177f in term 1.
I20260929 18:35:46.894193 1362 catalog_manager.cc:5495] T fc468d955e87421180f26c977695bb85 P 691302f4da3b41f8ac5c4bfbea14177f reported cstate change: term changed from 0 to 1, leader changed from <none> to 691302f4da3b41f8ac5c4bfbea14177f (127.0.198.3). New cstate: current_term: 1 leader_uuid: "691302f4da3b41f8ac5c4bfbea14177f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45f1cd02438f8b5fc254bb861169" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691302f4da3b41f8ac5c4bfbea14177f" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce20c82ff8e4ca8b03713e461bd03dd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36495 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:35:46.920248 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.911004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.0.198.1:36495)
0929 18:35:46.911171 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.911184 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.911275 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:35:46.911382 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.911384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.911425 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.911497 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.912165 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.912728 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.913626 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.913837 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.913918 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.916462 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.916475 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.916535 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.916561 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.918474 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.918955 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.918960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.918961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.919056 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.919336 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.919339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.919340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.919645 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.919817 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.919981 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.920093 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:35:46.946761 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.937842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.0.198.1:36495)
0929 18:35:46.938018 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.938023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.938099 (+ 76us) server_negotiation.cc:407] Connection header received
0929 18:35:46.938252 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.938256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.938299 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.938365 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.939105 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.939655 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.940494 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.940706 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.940791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.943315 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.943319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.943320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.943348 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.944807 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.945270 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.945272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.945274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.945361 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.945786 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.945789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.945790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.945999 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.946111 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.946396 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.946599 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:35:46 dist-test-slave-q5qx krb5kdc[1317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:35:46.972638 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.964181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.0.198.1:36495)
0929 18:35:46.964321 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.964325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.964512 (+ 187us) server_negotiation.cc:407] Connection header received
0929 18:35:46.964652 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.964654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.964698 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.964775 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.965422 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.965932 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.966856 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.967079 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.967160 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.969552 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.969555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.969557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.969581 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.970895 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.971389 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.971392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.971394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.971483 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.971745 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.971747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.971749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.971926 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.972034 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.972341 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.972474 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 18:35:46.974159 1467 tablet_service.cc:2908] Rejecting scan request for tablet fc468d955e87421180f26c977695bb85: Uninitialized: clean time has not yet been initialized
I20260929 18:35:46.976290 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1424
I20260929 18:35:46.981367 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1558
I20260929 18:35:46.986344 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1692
I20260929 18:35:46.990901 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1330
2026-09-29T18:35:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2160 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:47 dist-test-slave-q5qx krb5kdc[1862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:47 dist-test-slave-q5qx krb5kdc[1862](info): set up 2 sockets
Sep 29 18:35:47 dist-test-slave-q5qx krb5kdc[1862](info): commencing operation
krb5kdc: starting...
W20260929 18:35:47.953487 792 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:35:48.073292 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35413
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:35413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:48.181905 1875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:48.182214 1875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:48.182260 1875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:48.186257 1875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:48.186316 1875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:35:48.186332 1875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:48.186348 1875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:48.186368 1875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:48.186396 1875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:48.191727 1875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:35413
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:48.192760 1875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:48.193732 1875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:48.202137 1881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:48.202149 1883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:48.202370 1875 server_base.cc:1034] running on GCE node
W20260929 18:35:48.202137 1880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:48.203099 1875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:48.204201 1875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:48.205368 1875 hybrid_clock.cc:648] HybridClock initialized: now 1790706948205340 us; error 51 us; skew 500 ppm
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:48.208178 1875 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:35:48.209074 1875 webserver.cc:460] Webserver started at http://127.0.198.62:46357/ using document root <none> and password file <none>
I20260929 18:35:48.209615 1875 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:48.209681 1875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:48.209859 1875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:48.211930 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "5769a1c2896d46b7b6b3fe3c87c02e73"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "584a3972b8a7339db652eaf16bbdd17f"
server_key_iv: "799c569fda8231910422f30c6a59042c"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.212468 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "5769a1c2896d46b7b6b3fe3c87c02e73"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "584a3972b8a7339db652eaf16bbdd17f"
server_key_iv: "799c569fda8231910422f30c6a59042c"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.216249 1875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:48.218719 1890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:48.220081 1875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:48.220199 1875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "5769a1c2896d46b7b6b3fe3c87c02e73"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "584a3972b8a7339db652eaf16bbdd17f"
server_key_iv: "799c569fda8231910422f30c6a59042c"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.220304 1875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:48.241307 1875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:48.244710 1875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:48.244916 1875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:48.252568 1875 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:35413
I20260929 18:35:48.252604 1952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:35413 every 8 connection(s)
I20260929 18:35:48.253544 1875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:48.256388 1953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:48.259197 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1875
I20260929 18:35:48.259301 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:35:48.259555 792 external_mini_cluster.cc:1428] Setting key 72601358928d19b79c78c0db4197fb55
I20260929 18:35:48.262526 1953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73: Bootstrap starting.
I20260929 18:35:48.265275 1953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:48.266078 1953 log.cc:824] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:48.268059 1953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73: No bootstrap required, opened a new log
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:48.271756 1953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35413 } }
I20260929 18:35:48.272017 1953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:48.272091 1953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5769a1c2896d46b7b6b3fe3c87c02e73, State: Initialized, Role: FOLLOWER
I20260929 18:35:48.272575 1953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35413 } }
I20260929 18:35:48.272713 1953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:48.272778 1953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:48.272883 1953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:48.273833 1953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35413 } }
I20260929 18:35:48.274173 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.260839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.0.198.62:35413)
0929 18:35:48.261293 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:48.261312 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:48.261353 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:35:48.262170 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:48.262193 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:48.262616 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:48.262996 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:48.263673 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.265165 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.266236 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.266751 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.266902 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.269291 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:48.269297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:48.269312 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:48.269343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:48.271182 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.271668 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.271674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.271678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.271796 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.272061 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.272065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.272066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.272497 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:48.272705 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:48.272968 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 18:35:48.273271 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
I20260929 18:35:48.274185 1953 leader_election.cc:302] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5769a1c2896d46b7b6b3fe3c87c02e73; no voters:
I20260929 18:35:48.274560 1953 leader_election.cc:288] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:48.274652 1958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:48.274813 1958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [term 1 LEADER]: Becoming Leader. State: Replica: 5769a1c2896d46b7b6b3fe3c87c02e73, State: Running, Role: LEADER
I20260929 18:35:48.275182 1958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35413 } }
I20260929 18:35:48.275876 1953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:48.276952 1959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35413 } } }
I20260929 18:35:48.277092 1959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:48.277272 1960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5769a1c2896d46b7b6b3fe3c87c02e73. Latest consensus state: current_term: 1 leader_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5769a1c2896d46b7b6b3fe3c87c02e73" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35413 } } }
I20260929 18:35:48.277452 1960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:48.277805 1971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:48.280838 1971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:48.286773 1971 catalog_manager.cc:1347] Generated new cluster ID: 296a1eb6200d481488aeff53b0b879c0
I20260929 18:35:48.286926 1971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:48.321523 1971 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:48.323109 1971 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:48.332000 1971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73: Generated new TSK 0
I20260929 18:35:48.332883 1971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:35:48.363444 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35413
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:48.476217 1979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:48.476506 1979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:48.476559 1979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:48.480824 1979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:48.480891 1979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:48.480931 1979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:35:48.480966 1979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:35:48.486395 1979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:48.487526 1979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:48.488567 1979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:48.498230 1984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:48.498404 1987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:48.498397 1979 server_base.cc:1034] running on GCE node
W20260929 18:35:48.498363 1985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:48.499027 1979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:48.499682 1979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:48.500852 1979 hybrid_clock.cc:648] HybridClock initialized: now 1790706948500823 us; error 50 us; skew 500 ppm
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:48.503657 1979 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:35:48.504578 1979 webserver.cc:460] Webserver started at http://127.0.198.1:41275/ using document root <none> and password file <none>
I20260929 18:35:48.505174 1979 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:48.505240 1979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:48.505422 1979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:48.507565 1979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "bd1958ad61444424b0698e00974e0ab2"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "2f2820d473a17d2a49342104af1c7852"
server_key_iv: "643014e6ab4b87778a440db514c528ff"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.508100 1979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "bd1958ad61444424b0698e00974e0ab2"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "2f2820d473a17d2a49342104af1c7852"
server_key_iv: "643014e6ab4b87778a440db514c528ff"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.511798 1979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:35:48.514209 1994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:48.515648 1979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:48.515770 1979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "bd1958ad61444424b0698e00974e0ab2"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "2f2820d473a17d2a49342104af1c7852"
server_key_iv: "643014e6ab4b87778a440db514c528ff"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.515882 1979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:48.534817 1979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:48.538290 1979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:48.538472 1979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:48.540004 1979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:48.540057 1979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:48.540112 1979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:48.540138 1979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:48.550089 1979 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:41387
I20260929 18:35:48.550192 2109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:41387 every 8 connection(s)
I20260929 18:35:48.551182 1979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:48.553418 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.540948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:47499 (local address 127.0.198.62:35413)
0929 18:35:48.541090 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:48.541094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:48.541985 (+ 891us) server_negotiation.cc:407] Connection header received
0929 18:35:48.543035 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:48.543039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:48.543101 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:48.543207 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:48.544413 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.545220 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.546554 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.546909 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.547023 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.549601 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:48.549606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:48.549609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:48.549638 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:48.551381 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.551984 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.551988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.551990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.552082 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.552388 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.552391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.552392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.552596 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:48.552675 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:48.553193 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 18:35:48.553300 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:35:48.554344 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.541325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35413 (local address 127.0.198.1:47499)
0929 18:35:48.541778 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:48.541828 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:48.542793 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:48.543397 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:48.543405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:48.543764 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:48.544272 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.544286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.545356 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.545360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.546449 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.546459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.547065 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.547069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.547245 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.547881 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:48.547902 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:48.549419 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:48.551490 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.551495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.551507 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.551826 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.552174 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.552176 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.552178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.552308 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.552708 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:48.552714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:48.552953 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 18:35:48.553144 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 18:35:48.553398 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260929 18:35:48.555727 2110 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35413
I20260929 18:35:48.555859 2110 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:48.556353 2110 heartbeater.cc:511] Master 127.0.198.62:35413 requested a full tablet report, sending...
I20260929 18:35:48.557909 1902 ts_manager.cc:194] Registered new tserver with Master: bd1958ad61444424b0698e00974e0ab2 (127.0.198.1:41387)
I20260929 18:35:48.559103 1902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:47499
I20260929 18:35:48.560341 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 1979
I20260929 18:35:48.560449 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:48.560717 792 external_mini_cluster.cc:1428] Setting key 05020afe598b5700631e0b2e85365278
I20260929 18:35:48.565011 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.557850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35413 (local address 127.0.198.1:55773)
0929 18:35:48.558149 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:48.558161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:48.558258 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:48.558545 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:48.558548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:48.558727 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:48.558827 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.558832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.559644 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.559647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.560314 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.560324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.560732 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.560736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.560823 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.561323 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:48.561337 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:48.561726 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:48.563416 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.563420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.563432 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.563663 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.563962 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.563965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.563966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.564054 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.564447 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:48.564450 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:48.564590 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 18:35:48.564681 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:35:48.564848 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":136,"threads_started":1}
I20260929 18:35:48.565016 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.558008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:55773 (local address 127.0.198.62:35413)
0929 18:35:48.558173 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:48.558178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:48.558233 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:35:48.558332 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:48.558335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:48.558378 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:48.558440 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:48.558942 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.559518 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.560425 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.560637 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.560720 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.561846 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:48.561850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:48.561852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:48.561877 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:48.563295 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.563761 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.563764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.563766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.563855 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.564140 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.564144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.564146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.564350 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:48.564452 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:48.564746 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:35:48.564860 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:35:48.588883 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35413
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:48.699698 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:48.699975 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:48.700017 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:48.704039 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:48.704106 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:48.704146 2119 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:35:48.704180 2119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:35:48.709527 2119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:48.710543 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:48.711472 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:48.721338 2124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:48.721289 2125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:48.721310 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:48.721369 2119 server_base.cc:1034] running on GCE node
I20260929 18:35:48.721917 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:48.722692 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:48.723878 2119 hybrid_clock.cc:648] HybridClock initialized: now 1790706948723865 us; error 37 us; skew 500 ppm
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:48.726856 2119 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:35:48.727845 2119 webserver.cc:460] Webserver started at http://127.0.198.2:42247/ using document root <none> and password file <none>
I20260929 18:35:48.728417 2119 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:48.728479 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:48.728690 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:48.730857 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "e4f62fd230ae4c47b9305a14a0c8351f"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "a14cbba16dbeb2a00bec441c132511ef"
server_key_iv: "1a6054bfa55be761cae50f9c6b532b82"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.731400 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "e4f62fd230ae4c47b9305a14a0c8351f"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "a14cbba16dbeb2a00bec441c132511ef"
server_key_iv: "1a6054bfa55be761cae50f9c6b532b82"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.735181 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 18:35:48.737674 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:48.739039 2119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:35:48.739156 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "e4f62fd230ae4c47b9305a14a0c8351f"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "a14cbba16dbeb2a00bec441c132511ef"
server_key_iv: "1a6054bfa55be761cae50f9c6b532b82"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.739275 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:48.771168 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:48.774394 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:48.774562 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:48.776130 2119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:48.776181 2119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:48.776223 2119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:48.776249 2119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:48.786309 2119 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:35391
I20260929 18:35:48.786338 2249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:35391 every 8 connection(s)
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:48.787251 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:48.790992 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.777026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:42471 (local address 127.0.198.62:35413)
0929 18:35:48.777139 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:48.777142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:48.778392 (+ 1250us) server_negotiation.cc:407] Connection header received
0929 18:35:48.779451 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:48.779454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:48.779596 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:48.779711 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:48.781369 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.782302 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.784045 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.784274 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.784386 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.787078 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:48.787081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:48.787083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:48.787110 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:48.788687 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.789199 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.789202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.789204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.789283 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.789687 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.789690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.789691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.789885 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:48.789961 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:48.790746 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 18:35:48.790855 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:35:48.791858 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.777653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35413 (local address 127.0.198.2:42471)
0929 18:35:48.778247 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:48.778294 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:48.779242 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:48.779974 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:48.779983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:48.780466 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:48.781197 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.781218 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.782451 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.782455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.783895 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.783908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.784572 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.784575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.784722 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.785322 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:48.785343 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:48.786897 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:48.788823 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.788828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.788841 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.789102 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.789393 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.789396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.789397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.789599 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.790030 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:48.790038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:48.790373 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 18:35:48.790669 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 18:35:48.790952 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":133,"threads_started":1}
I20260929 18:35:48.793088 2250 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35413
I20260929 18:35:48.793234 2250 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:48.793721 2250 heartbeater.cc:511] Master 127.0.198.62:35413 requested a full tablet report, sending...
I20260929 18:35:48.794718 1907 ts_manager.cc:194] Registered new tserver with Master: e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2:35391)
I20260929 18:35:48.795382 1907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:42471
I20260929 18:35:48.795459 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 2119
I20260929 18:35:48.795567 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:48.795845 792 external_mini_cluster.cc:1428] Setting key 8b66918b4794988a21c66e36390f3bc5
I20260929 18:35:48.802392 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.795273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:33537 (local address 127.0.198.62:35413)
0929 18:35:48.795372 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:48.795376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:48.795573 (+ 197us) server_negotiation.cc:407] Connection header received
0929 18:35:48.795740 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:48.795743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:48.795786 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:48.795833 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:48.796416 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.797085 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.797848 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.798114 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.798193 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.799332 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:48.799335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:48.799341 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:48.799362 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:48.800737 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.801211 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.801214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.801215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.801301 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:48.801590 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:48.801592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:48.801594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:48.801801 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:48.801875 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:48.802179 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 18:35:48.802276 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:35:48.802476 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:48.795185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35413 (local address 127.0.198.2:33537)
0929 18:35:48.795495 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:48.795509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:48.795638 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:48.795976 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:48.795979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:48.796175 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:48.796318 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.796327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.797206 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.797209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.797748 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:48.797755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:48.798213 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:48.798216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:48.798302 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:48.798759 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:48.798772 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:48.799189 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:48.800843 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.800847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.800849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.801097 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.801391 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:48.801395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:48.801397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:48.801490 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:48.801887 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:48.801899 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:48.802022 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 18:35:48.802135 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:35:48.802281 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:35:48.825323 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35413
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:48.936021 2259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:48.936304 2259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:48.936349 2259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:48.940459 2259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:48.940527 2259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:48.940567 2259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:35:48.940606 2259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:35:48.946081 2259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:48.947153 2259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:48.948125 2259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:48.957630 2264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:48.957646 2265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:48.957649 2267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:48.958036 2259 server_base.cc:1034] running on GCE node
I20260929 18:35:48.958419 2259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:48.959051 2259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:48.960206 2259 hybrid_clock.cc:648] HybridClock initialized: now 1790706948960185 us; error 33 us; skew 500 ppm
Sep 29 18:35:48 dist-test-slave-q5qx krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:48.963068 2259 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:35:48.963963 2259 webserver.cc:460] Webserver started at http://127.0.198.3:37049/ using document root <none> and password file <none>
I20260929 18:35:48.964509 2259 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:48.964569 2259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:48.964735 2259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:48.966883 2259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "bdfc7aff1f8b43bc97da36480faf3119"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "b5bed5009414e3963b03b53647e8e147"
server_key_iv: "037ad1de3706261ad74062ee3e755027"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.967408 2259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "bdfc7aff1f8b43bc97da36480faf3119"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "b5bed5009414e3963b03b53647e8e147"
server_key_iv: "037ad1de3706261ad74062ee3e755027"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.971235 2259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:35:48.973786 2274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:48.975076 2259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:48.975193 2259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "bdfc7aff1f8b43bc97da36480faf3119"
format_stamp: "Formatted at 2026-09-29 18:35:48 on dist-test-slave-q5qx"
server_key: "b5bed5009414e3963b03b53647e8e147"
server_key_iv: "037ad1de3706261ad74062ee3e755027"
server_key_version: "encryptionkey@0"
I20260929 18:35:48.975294 2259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:49.011123 2259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:49.014797 2259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:49.015023 2259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:49.016702 2259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:49.016759 2259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:49.016813 2259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:49.016841 2259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:49.026739 2259 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:35493
I20260929 18:35:49.026842 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:35493 every 8 connection(s)
I20260929 18:35:49.027695 2259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:49.031932 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 2259
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:49.032037 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:49.032290 792 external_mini_cluster.cc:1428] Setting key 9f94ff2abe3ec9bc11299f1c6dc2cb6d
I20260929 18:35:49.036424 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.017629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:60845 (local address 127.0.198.62:35413)
0929 18:35:49.017770 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.017775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.018674 (+ 899us) server_negotiation.cc:407] Connection header received
0929 18:35:49.019868 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.019871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.019934 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.020069 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:49.021305 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.022153 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.024045 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.024317 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.024408 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.032720 (+ 8312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:49.032724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:49.032726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:49.032754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:49.034374 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.034888 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.034892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.034894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.034980 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.035294 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.035298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.035299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.035517 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:49.035602 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.036146 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.036267 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:49.037253 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.018057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35413 (local address 127.0.198.3:60845)
0929 18:35:49.018508 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.018557 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.019637 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.020219 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.020226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.020656 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:49.021149 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.021164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.022318 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.022322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.023846 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.023857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.024485 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.024490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.024705 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.030866 (+ 6161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:49.030886 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:49.032510 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:49.034504 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:49.034512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:49.034531 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:49.034795 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:49.035060 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:49.035064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:49.035066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:49.035212 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:49.035617 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:49.035625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:49.035915 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.036092 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.036433 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260929 18:35:49.038405 2390 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35413
I20260929 18:35:49.038631 2390 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:49.039233 2390 heartbeater.cc:511] Master 127.0.198.62:35413 requested a full tablet report, sending...
I20260929 18:35:49.040313 1902 ts_manager.cc:194] Registered new tserver with Master: bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3:35493)
I20260929 18:35:49.040946 1902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:60845
I20260929 18:35:49.046433 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:49.048066 2396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.040788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35413 (local address 127.0.198.3:50093)
0929 18:35:49.041164 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.041178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.041275 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.041481 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.041483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.041673 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:49.041771 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.041777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.042848 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.042851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.043443 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.043453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.043898 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.043901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.043974 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.044479 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:49.044491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:49.044872 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:49.046634 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:49.046638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:49.046641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:49.046854 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:49.047095 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:49.047098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:49.047100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:49.047175 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:49.047504 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:49.047507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:49.047594 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.047676 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.047821 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":112,"threads_started":1}
I20260929 18:35:49.048054 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.040887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:50093 (local address 127.0.198.62:35413)
0929 18:35:49.041007 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.041012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.041247 (+ 235us) server_negotiation.cc:407] Connection header received
0929 18:35:49.041352 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.041355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.041393 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.041452 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:49.041872 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.042705 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.043575 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.043791 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.043866 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.044995 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:49.044998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:49.045001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:49.045022 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:49.046492 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.046939 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.046942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.046944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.047020 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.047249 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.047252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.047254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.047424 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:49.047506 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.047734 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.047831 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:49.081351 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.072175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.0.198.62:35413)
0929 18:35:49.072314 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.072318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.072338 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:35:49.072393 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.072396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.072449 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.072556 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:49.073620 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.074328 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.075334 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.075690 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.075796 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.078397 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:49.078400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:49.078402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:49.078431 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:49.079813 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.080246 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.080249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.080250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.080341 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.080610 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.080613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.080614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.080839 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:49.080951 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.081130 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.081230 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:35:49.084335 1902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:49.087380 1902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:49.101497 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.096929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.0.198.1:41387)
0929 18:35:49.097429 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.097437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.097460 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:35:49.097551 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.097557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.097744 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.097887 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.098191 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.099018 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.100262 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.100854 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.100954 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.101119 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.101262 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.101359 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":94,"threads_started":1}
I20260929 18:35:49.101504 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.096246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41387 (local address 127.0.0.1:41440)
0929 18:35:49.096575 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.096624 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.096759 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.097897 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.097903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.097925 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.098086 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.098092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.099122 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.099125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.100136 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.100144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.100966 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.100970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.101077 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.101112 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.101221 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.101349 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260929 18:35:49.102681 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.096323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35391 (local address 127.0.0.1:51166)
0929 18:35:49.097119 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.097132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.097218 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.098449 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.098452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.098469 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.098551 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.098557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.099935 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.099938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.101260 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.101267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.102265 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.102269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.102372 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.102393 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.102440 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.102512 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260929 18:35:49.102754 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.097033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.0.198.3:35493)
0929 18:35:49.097570 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.097582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.097792 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:35:49.097899 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.097905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.098146 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.098287 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.098677 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.099826 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.101281 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.102111 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.102222 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.102374 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.102481 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.102622 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":98,"threads_started":1}
I20260929 18:35:49.103313 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.097391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.0.198.2:35391)
0929 18:35:49.098051 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.098062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.098082 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:35:49.098145 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.098150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.098305 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.098463 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.098652 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.099809 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.101383 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.102146 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.102956 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.103077 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.103116 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.103188 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":93,"threads_started":1}
I20260929 18:35:49.104105 2408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.096503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:35493 (local address 127.0.0.1:33754)
0929 18:35:49.097706 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.097715 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.097797 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.098272 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.098275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.098292 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.098369 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.098375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.099935 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.099938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.101145 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.101154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.102229 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.102233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.102319 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.102376 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.102442 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.102515 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":44,"thread_start_us":83,"threads_started":1}
I20260929 18:35:49.104928 2044 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed6606e9e39449a9d933b7508c7356a (DEFAULT_TABLE table=test-table [id=4cff4980ad9e4290b031f4b4adb281eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:49.106155 2044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed6606e9e39449a9d933b7508c7356a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.106060 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed6606e9e39449a9d933b7508c7356a (DEFAULT_TABLE table=test-table [id=4cff4980ad9e4290b031f4b4adb281eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:49.107115 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed6606e9e39449a9d933b7508c7356a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.107111 2324 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed6606e9e39449a9d933b7508c7356a (DEFAULT_TABLE table=test-table [id=4cff4980ad9e4290b031f4b4adb281eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:49.108036 2324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed6606e9e39449a9d933b7508c7356a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.114478 2413 tablet_bootstrap.cc:492] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Bootstrap starting.
I20260929 18:35:49.115056 2414 tablet_bootstrap.cc:492] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Bootstrap starting.
I20260929 18:35:49.115887 2415 tablet_bootstrap.cc:492] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Bootstrap starting.
I20260929 18:35:49.117002 2413 tablet_bootstrap.cc:654] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.117002 2414 tablet_bootstrap.cc:654] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.117553 2415 tablet_bootstrap.cc:654] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.117897 2414 log.cc:824] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:49.118140 2413 log.cc:824] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:49.118379 2415 log.cc:824] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:49.119941 2414 tablet_bootstrap.cc:492] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: No bootstrap required, opened a new log
I20260929 18:35:49.120129 2413 tablet_bootstrap.cc:492] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: No bootstrap required, opened a new log
I20260929 18:35:49.120142 2414 ts_tablet_manager.cc:1397] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:35:49.120316 2413 ts_tablet_manager.cc:1397] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:35:49.121241 2415 tablet_bootstrap.cc:492] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: No bootstrap required, opened a new log
I20260929 18:35:49.121438 2415 ts_tablet_manager.cc:1397] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 18:35:49.123507 2414 raft_consensus.cc:348] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.123802 2414 raft_consensus.cc:374] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.123857 2414 raft_consensus.cc:729] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdfc7aff1f8b43bc97da36480faf3119, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.124351 2414 consensus_queue.cc:260] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.125069 2390 heartbeater.cc:503] Master 127.0.198.62:35413 was elected leader, sending a full tablet report...
I20260929 18:35:49.125425 2414 ts_tablet_manager.cc:1428] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:35:49.125602 2415 raft_consensus.cc:348] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.125873 2415 raft_consensus.cc:374] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.125926 2415 raft_consensus.cc:729] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f62fd230ae4c47b9305a14a0c8351f, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.126396 2415 consensus_queue.cc:260] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.126447 2413 raft_consensus.cc:348] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.126710 2413 raft_consensus.cc:374] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.126771 2413 raft_consensus.cc:729] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1958ad61444424b0698e00974e0ab2, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.127161 2250 heartbeater.cc:503] Master 127.0.198.62:35413 was elected leader, sending a full tablet report...
I20260929 18:35:49.127185 2413 consensus_queue.cc:260] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.127442 2415 ts_tablet_manager.cc:1428] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 18:35:49.128062 2110 heartbeater.cc:503] Master 127.0.198.62:35413 was elected leader, sending a full tablet report...
I20260929 18:35:49.128077 2413 ts_tablet_manager.cc:1428] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260929 18:35:49.167196 2421 raft_consensus.cc:482] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:49.167403 2421 raft_consensus.cc:504] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.168890 2421 leader_election.cc:288] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2:35391), bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3:35493)
I20260929 18:35:49.174180 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.169294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:35141 (local address 127.0.198.2:35391)
0929 18:35:49.169456 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.169461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.169483 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:35:49.169659 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.169662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.169716 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.169793 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.170216 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.171007 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.172744 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.173334 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.173440 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.173523 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.173935 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.174047 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:35:49.174208 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.169873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:44313 (local address 127.0.198.3:35493)
0929 18:35:49.170016 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.170021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.170033 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:35:49.170081 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.170084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.170128 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.170190 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.170539 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.171068 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.172764 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.173361 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.173467 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.173530 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.173922 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.174032 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:49.174575 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.169508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:35493 (local address 127.0.198.1:44313)
0929 18:35:49.169800 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.169813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.169895 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.170279 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.170281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.170294 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.170396 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.170403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.171189 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.171191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.172620 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.172630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.173591 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.173594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.173756 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.173809 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.174341 (+ 532us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.174421 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":145,"thread_start_us":90,"threads_started":1}
I20260929 18:35:49.175051 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.169116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35391 (local address 127.0.198.1:35141)
0929 18:35:49.169407 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.169423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.169546 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.169910 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.169913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.169932 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.170093 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.170100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.171141 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.171144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.172621 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.172630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.173589 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.173592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.173779 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.173813 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.174813 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.174899 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":116,"thread_start_us":80,"threads_started":1}
I20260929 18:35:49.175258 2344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6606e9e39449a9d933b7508c7356a" candidate_uuid: "bd1958ad61444424b0698e00974e0ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdfc7aff1f8b43bc97da36480faf3119" is_pre_election: true
I20260929 18:35:49.175613 2344 raft_consensus.cc:2454] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1958ad61444424b0698e00974e0ab2 in term 0.
I20260929 18:35:49.176316 1998 leader_election.cc:302] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1958ad61444424b0698e00974e0ab2, bdfc7aff1f8b43bc97da36480faf3119; no voters:
I20260929 18:35:49.176445 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6606e9e39449a9d933b7508c7356a" candidate_uuid: "bd1958ad61444424b0698e00974e0ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" is_pre_election: true
I20260929 18:35:49.176728 2204 raft_consensus.cc:2454] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1958ad61444424b0698e00974e0ab2 in term 0.
I20260929 18:35:49.177048 2421 raft_consensus.cc:2781] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:49.177142 2421 raft_consensus.cc:482] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:49.177192 2421 raft_consensus.cc:3037] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.178367 2421 raft_consensus.cc:504] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.178933 2421 leader_election.cc:288] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [CANDIDATE]: Term 1 election: Requested vote from peers e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2:35391), bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3:35493)
I20260929 18:35:49.179032 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6606e9e39449a9d933b7508c7356a" candidate_uuid: "bd1958ad61444424b0698e00974e0ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f62fd230ae4c47b9305a14a0c8351f"
I20260929 18:35:49.179150 2204 raft_consensus.cc:3037] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.180480 2204 raft_consensus.cc:2454] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1958ad61444424b0698e00974e0ab2 in term 1.
I20260929 18:35:49.180804 2344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6606e9e39449a9d933b7508c7356a" candidate_uuid: "bd1958ad61444424b0698e00974e0ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdfc7aff1f8b43bc97da36480faf3119"
I20260929 18:35:49.180872 1996 leader_election.cc:302] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1958ad61444424b0698e00974e0ab2, e4f62fd230ae4c47b9305a14a0c8351f; no voters:
I20260929 18:35:49.180935 2344 raft_consensus.cc:3037] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.181049 2421 raft_consensus.cc:2781] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:49.181317 2421 raft_consensus.cc:686] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 1 LEADER]: Becoming Leader. State: Replica: bd1958ad61444424b0698e00974e0ab2, State: Running, Role: LEADER
I20260929 18:35:49.181661 2421 consensus_queue.cc:237] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.182317 2344 raft_consensus.cc:2454] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1958ad61444424b0698e00974e0ab2 in term 1.
I20260929 18:35:49.185292 1907 catalog_manager.cc:5495] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1958ad61444424b0698e00974e0ab2 (127.0.198.1). New cstate: current_term: 1 leader_uuid: "bd1958ad61444424b0698e00974e0ab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:49.207656 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.203955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.0.198.1:41387)
0929 18:35:49.204079 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.204082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.204097 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:35:49.204276 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.204278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.204334 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.204420 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:49.204711 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.205205 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.206171 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.206388 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.206468 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.206842 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.207265 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.207345 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.207457 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.207541 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:49.213619 2344 raft_consensus.cc:1261] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Refusing update from remote peer bd1958ad61444424b0698e00974e0ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:49.213683 2204 raft_consensus.cc:1261] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Refusing update from remote peer bd1958ad61444424b0698e00974e0ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:49.214501 2424 consensus_queue.cc:1035] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:49.214821 2421 consensus_queue.cc:1035] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:49.224355 2429 mvcc.cc:205] Tried to move back new op lower bound from 7334735663971979264 to 7334735663851139072. Current Snapshot: MvccSnapshot[applied={T|T < 7334735663971979264}]
I20260929 18:35:49.237545 1907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:49.237749 1907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:49.240584 1907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73: Sending DeleteTablet for 3 replicas of tablet 9ed6606e9e39449a9d933b7508c7356a
I20260929 18:35:49.241480 2044 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed6606e9e39449a9d933b7508c7356a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:49 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:41440
I20260929 18:35:49.241496 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed6606e9e39449a9d933b7508c7356a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:49 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:51166
I20260929 18:35:49.242039 2324 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed6606e9e39449a9d933b7508c7356a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:49 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33754
I20260929 18:35:49.242448 2437 tablet_replica.cc:330] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: stopping tablet replica
I20260929 18:35:49.242432 1907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:35:49.242551 2439 tablet_replica.cc:330] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: stopping tablet replica
I20260929 18:35:49.242554 2438 tablet_replica.cc:330] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: stopping tablet replica
W20260929 18:35:49.242874 1907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:49.242926 2439 raft_consensus.cc:2229] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:49.243268 2439 raft_consensus.cc:2258] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:49.243268 2437 raft_consensus.cc:2229] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:49.243536 2438 raft_consensus.cc:2229] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:49.243877 2437 raft_consensus.cc:2258] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:49.244823 2438 raft_consensus.cc:2258] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:49.246397 2439 ts_tablet_manager.cc:1905] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:49.246614 2437 ts_tablet_manager.cc:1905] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:49.248467 2324 tablet_service.cc:1467] Processing CreateTablet for tablet 53567730deda4196a59ad79bbb5b38e6 (DEFAULT_TABLE table=test-table [id=b1c09cce13f84664a6d5c2fe120ccb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:49.248768 2324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53567730deda4196a59ad79bbb5b38e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.248888 2044 tablet_service.cc:1467] Processing CreateTablet for tablet 53567730deda4196a59ad79bbb5b38e6 (DEFAULT_TABLE table=test-table [id=b1c09cce13f84664a6d5c2fe120ccb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:49.249142 2044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53567730deda4196a59ad79bbb5b38e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.249467 2439 ts_tablet_manager.cc:1918] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:49.249605 2439 log.cc:1196] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/9ed6606e9e39449a9d933b7508c7356a
I20260929 18:35:49.250002 2439 ts_tablet_manager.cc:1939] T 9ed6606e9e39449a9d933b7508c7356a P bdfc7aff1f8b43bc97da36480faf3119: Deleting consensus metadata
I20260929 18:35:49.251298 1894 catalog_manager.cc:4841] TS bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3:35493): tablet 9ed6606e9e39449a9d933b7508c7356a (table test-table [id=4cff4980ad9e4290b031f4b4adb281eb]) successfully deleted
I20260929 18:35:49.251427 2413 tablet_bootstrap.cc:492] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Bootstrap starting.
I20260929 18:35:49.251653 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 53567730deda4196a59ad79bbb5b38e6 (DEFAULT_TABLE table=test-table [id=b1c09cce13f84664a6d5c2fe120ccb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:49.251861 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53567730deda4196a59ad79bbb5b38e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.252391 2413 tablet_bootstrap.cc:654] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.253865 2438 ts_tablet_manager.cc:1905] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:49.253923 2414 tablet_bootstrap.cc:492] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Bootstrap starting.
I20260929 18:35:49.254860 2414 tablet_bootstrap.cc:654] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.256539 2415 tablet_bootstrap.cc:492] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Bootstrap starting.
I20260929 18:35:49.256659 2438 ts_tablet_manager.cc:1918] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:49.256780 2438 log.cc:1196] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/9ed6606e9e39449a9d933b7508c7356a
I20260929 18:35:49.257160 2438 ts_tablet_manager.cc:1939] T 9ed6606e9e39449a9d933b7508c7356a P bd1958ad61444424b0698e00974e0ab2: Deleting consensus metadata
I20260929 18:35:49.258026 2415 tablet_bootstrap.cc:654] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.258725 1893 catalog_manager.cc:4841] TS bd1958ad61444424b0698e00974e0ab2 (127.0.198.1:41387): tablet 9ed6606e9e39449a9d933b7508c7356a (table test-table [id=4cff4980ad9e4290b031f4b4adb281eb]) successfully deleted
I20260929 18:35:49.259270 2413 tablet_bootstrap.cc:492] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: No bootstrap required, opened a new log
I20260929 18:35:49.259351 2413 ts_tablet_manager.cc:1397] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260929 18:35:49.259500 2437 ts_tablet_manager.cc:1918] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:49.259585 2437 log.cc:1196] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/9ed6606e9e39449a9d933b7508c7356a
I20260929 18:35:49.259837 2437 ts_tablet_manager.cc:1939] T 9ed6606e9e39449a9d933b7508c7356a P e4f62fd230ae4c47b9305a14a0c8351f: Deleting consensus metadata
I20260929 18:35:49.259809 2413 raft_consensus.cc:348] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.259925 2413 raft_consensus.cc:374] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.259956 2413 raft_consensus.cc:729] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1958ad61444424b0698e00974e0ab2, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.260071 2413 consensus_queue.cc:260] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.260311 2413 ts_tablet_manager.cc:1428] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:35:49.261343 2415 tablet_bootstrap.cc:492] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: No bootstrap required, opened a new log
I20260929 18:35:49.261441 2415 ts_tablet_manager.cc:1397] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260929 18:35:49.261770 2414 tablet_bootstrap.cc:492] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: No bootstrap required, opened a new log
I20260929 18:35:49.261849 2414 ts_tablet_manager.cc:1397] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 18:35:49.261873 2415 raft_consensus.cc:348] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.261991 2415 raft_consensus.cc:374] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.262025 2415 raft_consensus.cc:729] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f62fd230ae4c47b9305a14a0c8351f, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.262135 2415 consensus_queue.cc:260] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.262372 2415 ts_tablet_manager.cc:1428] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:49.262326 2414 raft_consensus.cc:348] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.262422 2414 raft_consensus.cc:374] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.262451 2414 raft_consensus.cc:729] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdfc7aff1f8b43bc97da36480faf3119, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.262562 2414 consensus_queue.cc:260] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.262790 2414 ts_tablet_manager.cc:1428] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:35:49.263341 1892 catalog_manager.cc:4841] TS e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2:35391): tablet 9ed6606e9e39449a9d933b7508c7356a (table test-table [id=4cff4980ad9e4290b031f4b4adb281eb]) successfully deleted
W20260929 18:35:49.278636 2391 tablet.cc:2369] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:49.288758 2251 tablet.cc:2369] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:49.302554 2111 tablet.cc:2369] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:49.446499 2420 raft_consensus.cc:482] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:49.446664 2420 raft_consensus.cc:504] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.447947 2420 leader_election.cc:288] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd1958ad61444424b0698e00974e0ab2 (127.0.198.1:41387), bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3:35493)
I20260929 18:35:49.453212 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.448275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:59739 (local address 127.0.198.3:35493)
0929 18:35:49.448406 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.448410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.448631 (+ 221us) server_negotiation.cc:407] Connection header received
0929 18:35:49.448731 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.448734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.448784 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.448862 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.449219 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.450081 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.451830 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.452527 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.452614 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.452669 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.453003 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.453077 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:35:49.453212 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.448249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:53905 (local address 127.0.198.1:41387)
0929 18:35:49.448465 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.448468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.448591 (+ 123us) server_negotiation.cc:407] Connection header received
0929 18:35:49.448731 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.448734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.448784 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.448862 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.449219 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.450068 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.451836 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.452481 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.452604 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.452671 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.453002 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.453079 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 18:35:49.453277 2444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.448170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:35493 (local address 127.0.198.2:59739)
0929 18:35:49.448550 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.448561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.448637 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.448871 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.448874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.448909 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.449110 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.449122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.450179 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.450183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.451676 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.451688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.452652 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.452656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.452850 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.452888 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.452971 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.453091 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":185,"spinlock_wait_cycles":6400,"thread_start_us":76,"threads_started":1}
I20260929 18:35:49.453631 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.448119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41387 (local address 127.0.198.2:53905)
0929 18:35:49.448435 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.448519 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.448635 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.448871 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.448874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.448908 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.449114 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.449122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.450179 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.450183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.451679 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.451688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.452652 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.452656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.452845 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.452893 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.452971 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.453091 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":337,"spinlock_wait_cycles":19968,"thread_start_us":85,"threads_started":1}
I20260929 18:35:49.453724 2344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53567730deda4196a59ad79bbb5b38e6" candidate_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdfc7aff1f8b43bc97da36480faf3119" is_pre_election: true
I20260929 18:35:49.453892 2344 raft_consensus.cc:2454] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f62fd230ae4c47b9305a14a0c8351f in term 0.
I20260929 18:35:49.454159 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53567730deda4196a59ad79bbb5b38e6" candidate_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1958ad61444424b0698e00974e0ab2" is_pre_election: true
I20260929 18:35:49.454285 2138 leader_election.cc:302] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdfc7aff1f8b43bc97da36480faf3119, e4f62fd230ae4c47b9305a14a0c8351f; no voters:
I20260929 18:35:49.454365 2064 raft_consensus.cc:2454] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f62fd230ae4c47b9305a14a0c8351f in term 0.
I20260929 18:35:49.454535 2420 raft_consensus.cc:2781] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:49.454592 2420 raft_consensus.cc:482] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:49.454632 2420 raft_consensus.cc:3037] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.455557 2420 raft_consensus.cc:504] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.455894 2420 leader_election.cc:288] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [CANDIDATE]: Term 1 election: Requested vote from peers bd1958ad61444424b0698e00974e0ab2 (127.0.198.1:41387), bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3:35493)
I20260929 18:35:49.456184 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53567730deda4196a59ad79bbb5b38e6" candidate_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1958ad61444424b0698e00974e0ab2"
I20260929 18:35:49.456210 2344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53567730deda4196a59ad79bbb5b38e6" candidate_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdfc7aff1f8b43bc97da36480faf3119"
I20260929 18:35:49.456287 2064 raft_consensus.cc:3037] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.456311 2344 raft_consensus.cc:3037] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.457206 2064 raft_consensus.cc:2454] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f62fd230ae4c47b9305a14a0c8351f in term 1.
I20260929 18:35:49.457329 2344 raft_consensus.cc:2454] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f62fd230ae4c47b9305a14a0c8351f in term 1.
I20260929 18:35:49.457584 2138 leader_election.cc:302] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdfc7aff1f8b43bc97da36480faf3119, e4f62fd230ae4c47b9305a14a0c8351f; no voters:
I20260929 18:35:49.457727 2420 raft_consensus.cc:2781] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:49.457918 2420 raft_consensus.cc:686] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 1 LEADER]: Becoming Leader. State: Replica: e4f62fd230ae4c47b9305a14a0c8351f, State: Running, Role: LEADER
I20260929 18:35:49.458212 2420 consensus_queue.cc:237] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.460348 1907 catalog_manager.cc:5495] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f reported cstate change: term changed from 0 to 1, leader changed from <none> to e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2). New cstate: current_term: 1 leader_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:49.525853 2452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.511648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35413 (local address 127.0.0.1:44690)
0929 18:35:49.511902 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.511915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.512024 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.512229 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.512231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.512739 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:49.512832 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.512838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.513597 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.513600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.514221 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.514229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.514626 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.514628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.514682 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.521075 (+ 6393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:49.521088 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:49.522617 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:49.524237 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:49.524247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:49.524254 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:49.524588 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:49.524839 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:49.524841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:49.524843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:49.524915 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:49.525319 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:49.525324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:49.525432 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.525599 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.525721 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260929 18:35:49.525952 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.511750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.0.198.62:35413)
0929 18:35:49.511888 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.511893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.511983 (+ 90us) server_negotiation.cc:407] Connection header received
0929 18:35:49.512092 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.512094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.512138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.512204 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:49.512943 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.513470 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.514324 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.514538 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.514623 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.522773 (+ 8150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:49.522778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:49.522780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:49.522800 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:49.524122 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.524677 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.524681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.524682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.524768 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.525005 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.525008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.525010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.525205 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:49.525292 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.525700 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.525802 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:35:49.530051 1907 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:44690:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:35:49.530777 1907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:49.535596 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 0d14b64be8b24745b0175be81c21ef3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37c5fb4cdfcf4dabbfb1e180136d4a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:35:49.535596 2324 tablet_service.cc:1467] Processing CreateTablet for tablet 0d14b64be8b24745b0175be81c21ef3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37c5fb4cdfcf4dabbfb1e180136d4a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:35:49.535626 2044 tablet_service.cc:1467] Processing CreateTablet for tablet 0d14b64be8b24745b0175be81c21ef3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37c5fb4cdfcf4dabbfb1e180136d4a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:35:49.535988 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d14b64be8b24745b0175be81c21ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.535988 2324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d14b64be8b24745b0175be81c21ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.536001 2044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d14b64be8b24745b0175be81c21ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:49.538375 2413 tablet_bootstrap.cc:492] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: Bootstrap starting.
I20260929 18:35:49.538769 2414 tablet_bootstrap.cc:492] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: Bootstrap starting.
I20260929 18:35:49.538769 2415 tablet_bootstrap.cc:492] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: Bootstrap starting.
I20260929 18:35:49.539242 2413 tablet_bootstrap.cc:654] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.539791 2415 tablet_bootstrap.cc:654] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.540176 2414 tablet_bootstrap.cc:654] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:49.540356 2413 tablet_bootstrap.cc:492] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: No bootstrap required, opened a new log
I20260929 18:35:49.540444 2413 ts_tablet_manager.cc:1397] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:49.540978 2413 raft_consensus.cc:348] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.541129 2413 raft_consensus.cc:374] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.541168 2413 raft_consensus.cc:729] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1958ad61444424b0698e00974e0ab2, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.541288 2415 tablet_bootstrap.cc:492] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: No bootstrap required, opened a new log
I20260929 18:35:49.541291 2413 consensus_queue.cc:260] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.541366 2415 ts_tablet_manager.cc:1397] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:35:49.541546 2413 ts_tablet_manager.cc:1428] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:49.541757 2415 raft_consensus.cc:348] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.541857 2415 raft_consensus.cc:374] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.541893 2415 raft_consensus.cc:729] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f62fd230ae4c47b9305a14a0c8351f, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.541891 2432 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.542100 2432 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:49.542073 2415 consensus_queue.cc:260] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.542315 2415 ts_tablet_manager.cc:1428] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:49.542290 2420 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.542397 2420 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:49.542865 2414 tablet_bootstrap.cc:492] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: No bootstrap required, opened a new log
I20260929 18:35:49.542937 2414 ts_tablet_manager.cc:1397] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 18:35:49.543345 2414 raft_consensus.cc:348] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.543462 2414 raft_consensus.cc:374] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:49.543495 2414 raft_consensus.cc:729] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdfc7aff1f8b43bc97da36480faf3119, State: Initialized, Role: FOLLOWER
I20260929 18:35:49.543608 2414 consensus_queue.cc:260] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.544097 2419 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.544229 2419 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:49.544450 2414 ts_tablet_manager.cc:1428] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:35:49.753103 2419 raft_consensus.cc:482] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:49.753255 2419 raft_consensus.cc:504] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.754457 2419 leader_election.cc:288] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd1958ad61444424b0698e00974e0ab2 (127.0.198.1:41387), e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2:35391)
I20260929 18:35:49.758605 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.755038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:49409 (local address 127.0.198.1:41387)
0929 18:35:49.755152 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.755156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.755196 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:35:49.755362 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.755365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.755414 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.755509 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.755832 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.756326 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.757630 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.758074 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.758170 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.758231 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.758389 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.758462 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:35:49.758625 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.754719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35391 (local address 127.0.198.3:56939)
0929 18:35:49.755051 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.755065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.755177 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.755437 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.755440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.755456 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.755686 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.755692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.756442 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.756445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.757481 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.757490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.758161 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.758164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.758284 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.758302 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.758368 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.758451 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":178,"spinlock_wait_cycles":17536,"thread_start_us":70,"threads_started":1}
I20260929 18:35:49.758620 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.754799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:56939 (local address 127.0.198.2:35391)
0929 18:35:49.755121 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.755125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.755143 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:35:49.755286 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.755289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.755338 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.755413 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:49.755776 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.756334 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.757616 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.758057 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.758140 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.758205 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.758391 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.758454 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":75,"threads_started":1}
I20260929 18:35:49.758862 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.754722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41387 (local address 127.0.198.3:49409)
0929 18:35:49.755126 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.755138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.755247 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.755540 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.755543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.755587 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:49.755713 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.755720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.756441 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.756444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.757481 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.757491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.758175 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.758179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.758287 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.758304 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.758366 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.758448 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":99,"spinlock_wait_cycles":2816,"thread_start_us":75,"threads_started":1}
I20260929 18:35:49.759227 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d14b64be8b24745b0175be81c21ef3a" candidate_uuid: "bdfc7aff1f8b43bc97da36480faf3119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" is_pre_election: true
I20260929 18:35:49.759249 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d14b64be8b24745b0175be81c21ef3a" candidate_uuid: "bdfc7aff1f8b43bc97da36480faf3119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1958ad61444424b0698e00974e0ab2" is_pre_election: true
I20260929 18:35:49.759356 2204 raft_consensus.cc:2454] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdfc7aff1f8b43bc97da36480faf3119 in term 0.
I20260929 18:35:49.759373 2064 raft_consensus.cc:2454] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdfc7aff1f8b43bc97da36480faf3119 in term 0.
I20260929 18:35:49.759678 2276 leader_election.cc:302] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdfc7aff1f8b43bc97da36480faf3119, e4f62fd230ae4c47b9305a14a0c8351f; no voters:
I20260929 18:35:49.759922 2419 raft_consensus.cc:2781] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:49.759979 2419 raft_consensus.cc:482] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:49.760015 2419 raft_consensus.cc:3037] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.761245 2419 raft_consensus.cc:504] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.761587 2419 leader_election.cc:288] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [CANDIDATE]: Term 1 election: Requested vote from peers bd1958ad61444424b0698e00974e0ab2 (127.0.198.1:41387), e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2:35391)
I20260929 18:35:49.761904 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d14b64be8b24745b0175be81c21ef3a" candidate_uuid: "bdfc7aff1f8b43bc97da36480faf3119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1958ad61444424b0698e00974e0ab2"
I20260929 18:35:49.761902 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d14b64be8b24745b0175be81c21ef3a" candidate_uuid: "bdfc7aff1f8b43bc97da36480faf3119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f62fd230ae4c47b9305a14a0c8351f"
I20260929 18:35:49.762027 2064 raft_consensus.cc:3037] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.762130 2204 raft_consensus.cc:3037] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:49.762915 2064 raft_consensus.cc:2454] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdfc7aff1f8b43bc97da36480faf3119 in term 1.
I20260929 18:35:49.763113 2204 raft_consensus.cc:2454] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdfc7aff1f8b43bc97da36480faf3119 in term 1.
I20260929 18:35:49.763163 2277 leader_election.cc:302] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1958ad61444424b0698e00974e0ab2, bdfc7aff1f8b43bc97da36480faf3119; no voters:
I20260929 18:35:49.763302 2419 raft_consensus.cc:2781] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:49.763463 2419 raft_consensus.cc:686] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [term 1 LEADER]: Becoming Leader. State: Replica: bdfc7aff1f8b43bc97da36480faf3119, State: Running, Role: LEADER
I20260929 18:35:49.763705 2419 consensus_queue.cc:237] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } }
I20260929 18:35:49.764853 2459 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: TxnStatusTablet state changed. Reason: New leader bdfc7aff1f8b43bc97da36480faf3119. Latest consensus state: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.764958 2459 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:35:49.765153 2461 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:35:49.765290 2461 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:35:49.766129 1907 catalog_manager.cc:5495] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3). New cstate: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } health_report { overall_health: HEALTHY } } }
I20260929 18:35:49.782306 2452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.778122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:35493 (local address 127.0.0.1:33768)
0929 18:35:49.778249 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.778265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.778377 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.778694 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.778697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.778726 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:49.778848 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.778855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.779894 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.779896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.780527 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.780537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.781083 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.781085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.781151 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.781196 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.781958 (+ 762us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.782030 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.782107 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.782192 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 18:35:49.782327 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.778206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.198.3:35493)
0929 18:35:49.778331 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.778335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.778361 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:35:49.778502 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.778505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.778567 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.778670 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:49.778952 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.779785 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.780639 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.780979 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.781076 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.781317 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.781851 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.781963 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.782129 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.782188 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:35:49.785344 2204 raft_consensus.cc:1261] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Refusing update from remote peer bdfc7aff1f8b43bc97da36480faf3119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:49.785454 2064 raft_consensus.cc:1261] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Refusing update from remote peer bdfc7aff1f8b43bc97da36480faf3119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:49.785905 2419 consensus_queue.cc:1035] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:49.786473 2459 consensus_queue.cc:1035] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:49.787606 2420 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: TxnStatusTablet state changed. Reason: New leader bdfc7aff1f8b43bc97da36480faf3119. Latest consensus state: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.787737 2420 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:49.789733 2459 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.789842 2459 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:35:49.790143 2420 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.790246 2420 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P e4f62fd230ae4c47b9305a14a0c8351f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:49.791883 2432 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: TxnStatusTablet state changed. Reason: New leader bdfc7aff1f8b43bc97da36480faf3119. Latest consensus state: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.791996 2432 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:49.795727 2419 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.795859 2419 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P bdfc7aff1f8b43bc97da36480faf3119: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:35:49.796542 2432 tablet_replica.cc:439] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdfc7aff1f8b43bc97da36480faf3119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 } } peers { permanent_uuid: "e4f62fd230ae4c47b9305a14a0c8351f" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35391 } } peers { permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 } } }
I20260929 18:35:49.796661 2432 tablet_replica.cc:442] T 0d14b64be8b24745b0175be81c21ef3a P bd1958ad61444424b0698e00974e0ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:49.797340 2469 mvcc.cc:205] Tried to move back new op lower bound from 7334735666315468800 to 7334735666234429440. Current Snapshot: MvccSnapshot[applied={T|T < 7334735666315468800}]
I20260929 18:35:49.797497 2464 mvcc.cc:205] Tried to move back new op lower bound from 7334735666315468800 to 7334735666234429440. Current Snapshot: MvccSnapshot[applied={T|T < 7334735666315468800 or (T in {7334735666315468800})}]
I20260929 18:35:49.805831 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.801480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.0.198.2:35391)
0929 18:35:49.801581 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.801585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.801599 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:35:49.801664 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.801666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.801715 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.801808 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:49.802113 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.802888 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.804195 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.804545 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.804645 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.804896 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.805461 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.805575 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.805618 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.805688 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:35:49.817618 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.813262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:35493 (local address 127.0.198.2:47307)
0929 18:35:49.813529 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.813541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.813629 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.813843 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.813845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.813852 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:49.813939 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.813944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.815138 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.815141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.816318 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.816329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.816947 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.816950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.817029 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.817078 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.817350 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.817354 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.817398 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.817465 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260929 18:35:49.817608 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.813398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:47307 (local address 127.0.198.3:35493)
0929 18:35:49.813521 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.813525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.813590 (+ 65us) server_negotiation.cc:407] Connection header received
0929 18:35:49.813700 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.813703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.813751 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.813826 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:49.814097 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.815003 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.816453 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.816820 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.816926 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.817196 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.817280 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.817344 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.817410 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.817472 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:35:49.824846 2064 raft_consensus.cc:1261] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Refusing update from remote peer e4f62fd230ae4c47b9305a14a0c8351f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:49.824870 2344 raft_consensus.cc:1261] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Refusing update from remote peer e4f62fd230ae4c47b9305a14a0c8351f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:49.825276 2420 consensus_queue.cc:1035] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1958ad61444424b0698e00974e0ab2" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:49.825490 2445 consensus_queue.cc:1035] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdfc7aff1f8b43bc97da36480faf3119" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:49.847553 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.843578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:54043 (local address 127.0.198.2:35391)
0929 18:35:49.843696 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.843700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.843922 (+ 222us) server_negotiation.cc:407] Connection header received
0929 18:35:49.844093 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.844095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.844136 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.844193 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:49.844529 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.845058 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.846575 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.846802 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.846891 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.847185 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.847259 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.847307 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.847400 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.847442 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:35:49.847642 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.843475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35391 (local address 127.0.198.3:54043)
0929 18:35:49.843836 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:49.843851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:49.843957 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:49.844216 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:49.844219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:49.844227 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:49.844356 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.844362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.845265 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.845268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.846422 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:49.846434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.846917 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.846922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:49.847011 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.847068 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:49.847327 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:49.847332 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:35:49.847391 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:35:49.847480 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
I20260929 18:35:49.872841 1907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:49.873001 1907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:49.875005 1907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5769a1c2896d46b7b6b3fe3c87c02e73: Sending DeleteTablet for 3 replicas of tablet 53567730deda4196a59ad79bbb5b38e6
I20260929 18:35:49.875576 2044 tablet_service.cc:1514] Processing DeleteTablet for tablet 53567730deda4196a59ad79bbb5b38e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:49 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:41440
I20260929 18:35:49.875773 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 53567730deda4196a59ad79bbb5b38e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:49 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:51166
I20260929 18:35:49.875859 2324 tablet_service.cc:1514] Processing DeleteTablet for tablet 53567730deda4196a59ad79bbb5b38e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:49 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33754
I20260929 18:35:49.876312 2486 tablet_replica.cc:330] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: stopping tablet replica
I20260929 18:35:49.876495 2486 raft_consensus.cc:2229] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:49.876924 2486 raft_consensus.cc:2258] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:49.877729 2486 ts_tablet_manager.cc:1905] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:49.878420 2487 tablet_replica.cc:330] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: stopping tablet replica
I20260929 18:35:49.878551 2487 raft_consensus.cc:2229] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:49.878667 2487 raft_consensus.cc:2258] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:49.879311 2487 ts_tablet_manager.cc:1905] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:49.881094 2486 ts_tablet_manager.cc:1918] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:49.881219 2486 log.cc:1196] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/53567730deda4196a59ad79bbb5b38e6
I20260929 18:35:49.881547 2486 ts_tablet_manager.cc:1939] T 53567730deda4196a59ad79bbb5b38e6 P e4f62fd230ae4c47b9305a14a0c8351f: Deleting consensus metadata
I20260929 18:35:49.882014 2487 ts_tablet_manager.cc:1918] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:49.882100 2487 log.cc:1196] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/53567730deda4196a59ad79bbb5b38e6
I20260929 18:35:49.882313 2487 ts_tablet_manager.cc:1939] T 53567730deda4196a59ad79bbb5b38e6 P bdfc7aff1f8b43bc97da36480faf3119: Deleting consensus metadata
I20260929 18:35:49.882507 1892 catalog_manager.cc:4841] TS e4f62fd230ae4c47b9305a14a0c8351f (127.0.198.2:35391): tablet 53567730deda4196a59ad79bbb5b38e6 (table test-table [id=b1c09cce13f84664a6d5c2fe120ccb68]) successfully deleted
I20260929 18:35:49.882939 1894 catalog_manager.cc:4841] TS bdfc7aff1f8b43bc97da36480faf3119 (127.0.198.3:35493): tablet 53567730deda4196a59ad79bbb5b38e6 (table test-table [id=b1c09cce13f84664a6d5c2fe120ccb68]) successfully deleted
I20260929 18:35:49.886137 2485 tablet_replica.cc:330] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: stopping tablet replica
I20260929 18:35:49.886282 2485 raft_consensus.cc:2229] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:49.886401 2485 raft_consensus.cc:2258] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:49.887099 2485 ts_tablet_manager.cc:1905] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:35:49.890237 2485 ts_tablet_manager.cc:1918] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:49.890321 2485 log.cc:1196] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/53567730deda4196a59ad79bbb5b38e6
I20260929 18:35:49.890568 2485 ts_tablet_manager.cc:1939] T 53567730deda4196a59ad79bbb5b38e6 P bd1958ad61444424b0698e00974e0ab2: Deleting consensus metadata
I20260929 18:35:49.891304 1893 catalog_manager.cc:4841] TS bd1958ad61444424b0698e00974e0ab2 (127.0.198.1:41387): tablet 53567730deda4196a59ad79bbb5b38e6 (table test-table [id=b1c09cce13f84664a6d5c2fe120ccb68]) successfully deleted
I20260929 18:35:49.893316 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.877825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.0.198.1:41387)
0929 18:35:49.877938 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.877942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.883092 (+ 5150us) server_negotiation.cc:407] Connection header received
0929 18:35:49.883296 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.883299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.883349 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.883413 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:49.884051 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.884569 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.885359 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.885587 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.885673 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.888924 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:49.888934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:49.888947 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:49.888973 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:49.890976 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.891704 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.891711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.891713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.891807 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.892101 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.892105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.892106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.892418 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:49.892632 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.893012 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.893152 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 18:35:49.894315 2104 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41458
I20260929 18:35:49.905198 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:49.896637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.0.198.62:35413)
0929 18:35:49.896755 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:49.896759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:49.896907 (+ 148us) server_negotiation.cc:407] Connection header received
0929 18:35:49.897021 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:49.897024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:49.897073 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:49.897155 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:49.897899 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.898727 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.899876 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:49.900194 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:49.900296 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:49.901857 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:49.901861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:49.901864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:49.901892 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:49.903509 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.903980 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.903983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.903985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.904073 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:49.904333 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:49.904336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:49.904338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:49.904563 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:49.904692 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:49.904906 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:35:49.905030 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:35:49.905933 1907 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696
I20260929 18:35:49.907614 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1979
I20260929 18:35:49.916616 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 2119
I20260929 18:35:49.922469 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 2259
I20260929 18:35:49.928557 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 1875
2026-09-29T18:35:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2937 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[2494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[2494](info): set up 2 sockets
Sep 29 18:35:49 dist-test-slave-q5qx krb5kdc[2494](info): commencing operation
krb5kdc: starting...
W20260929 18:35:50.885648 792 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:35:50 dist-test-slave-q5qx krb5kdc[2494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:35:51.005786 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:37275
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:37275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:51.115892 2507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:51.116204 2507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:51.116259 2507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:51.120352 2507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:51.120421 2507 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:35:51.120445 2507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:51.120467 2507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:51.120487 2507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:51.120508 2507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:51.125945 2507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:37275
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:37275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:51.127157 2507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:51.128201 2507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:51.136850 2512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:51.136925 2515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:51.136850 2513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:51.137105 2507 server_base.cc:1034] running on GCE node
I20260929 18:35:51.137787 2507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:51.138947 2507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:51.140115 2507 hybrid_clock.cc:648] HybridClock initialized: now 1790706951140096 us; error 32 us; skew 500 ppm
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:51.143107 2507 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:35:51.144106 2507 webserver.cc:460] Webserver started at http://127.0.198.62:39707/ using document root <none> and password file <none>
I20260929 18:35:51.144685 2507 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:51.144753 2507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:51.144940 2507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:51.147120 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "813c6d3abbc94d7e94e9cd6624ec663a"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "b4734e52472dc91b757705d42718693b"
server_key_iv: "f1869f8833275656cb577b78f203da45"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.147678 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "813c6d3abbc94d7e94e9cd6624ec663a"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "b4734e52472dc91b757705d42718693b"
server_key_iv: "f1869f8833275656cb577b78f203da45"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.151777 2507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:51.154335 2522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:51.155700 2507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:35:51.155838 2507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "813c6d3abbc94d7e94e9cd6624ec663a"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "b4734e52472dc91b757705d42718693b"
server_key_iv: "f1869f8833275656cb577b78f203da45"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.155952 2507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:51.170931 2507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:51.174140 2507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:51.174321 2507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:51.181875 2507 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:37275
I20260929 18:35:51.181913 2584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:37275 every 8 connection(s)
I20260929 18:35:51.182852 2507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:51.185652 2585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:51.191291 2585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Bootstrap starting.
I20260929 18:35:51.192008 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 2507
I20260929 18:35:51.192128 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:35:51.192353 792 external_mini_cluster.cc:1428] Setting key 9e5964786d07e3315f5d2ffe0d324311
I20260929 18:35:51.194129 2585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:51.194850 2585 log.cc:824] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:51.196695 2585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: No bootstrap required, opened a new log
I20260929 18:35:51.200204 2585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37275 } }
I20260929 18:35:51.200441 2585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:51.200490 2585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813c6d3abbc94d7e94e9cd6624ec663a, State: Initialized, Role: FOLLOWER
I20260929 18:35:51.201016 2585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37275 } }
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:51.201159 2585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:51.201223 2585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:51.201319 2585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:51.202253 2585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37275 } }
I20260929 18:35:51.202610 2585 leader_election.cc:302] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 813c6d3abbc94d7e94e9cd6624ec663a; no voters:
I20260929 18:35:51.202934 2585 leader_election.cc:288] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:51.203078 2590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:51.203248 2590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [term 1 LEADER]: Becoming Leader. State: Replica: 813c6d3abbc94d7e94e9cd6624ec663a, State: Running, Role: LEADER
I20260929 18:35:51.203557 2590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37275 } }
I20260929 18:35:51.204262 2585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:51.204762 2590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813c6d3abbc94d7e94e9cd6624ec663a. Latest consensus state: current_term: 1 leader_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37275 } } }
I20260929 18:35:51.204885 2590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:51.205484 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813c6d3abbc94d7e94e9cd6624ec663a" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37275 } } }
I20260929 18:35:51.205595 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:51.207943 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.193560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.0.198.62:37275)
0929 18:35:51.194094 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.194111 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.194160 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:35:51.194876 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.194894 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.195264 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.195605 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.196356 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.197445 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.198373 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.198845 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.198989 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.201769 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.201774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.201787 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.201814 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.203891 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.204345 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.204352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.204359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.204476 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.204760 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.204763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.204764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.205455 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.206523 (+ 1068us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.206841 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.207114 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":145,"threads_started":1}
W20260929 18:35:51.209093 2605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:51.209174 2605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:51.209246 2595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:51.210029 2595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:51.215631 2595 catalog_manager.cc:1347] Generated new cluster ID: e649596a03a14d25ba375eba72b86cf0
I20260929 18:35:51.215704 2595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:51.222201 2595 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:51.223464 2595 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:51.238377 2595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Generated new TSK 0
I20260929 18:35:51.239269 2595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:35:51.269716 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37275
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:51.382743 2611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:51.383021 2611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:51.383087 2611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:51.387377 2611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:51.387457 2611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:51.387526 2611 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:35:51.387547 2611 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:35:51.387594 2611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:35:51.394631 2611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:51.395705 2611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:51.396677 2611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:51.406219 2616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:51.406322 2619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:51.406230 2617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:51.406450 2611 server_base.cc:1034] running on GCE node
I20260929 18:35:51.406925 2611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:51.407560 2611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:51.408749 2611 hybrid_clock.cc:648] HybridClock initialized: now 1790706951408705 us; error 58 us; skew 500 ppm
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:51.411602 2611 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:35:51.412468 2611 webserver.cc:460] Webserver started at http://127.0.198.1:35193/ using document root <none> and password file <none>
I20260929 18:35:51.413014 2611 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:51.413075 2611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:51.413275 2611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:51.415391 2611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "adde224930134115a5f3e6137cbb57e0"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "d494ac0d480a1c6913b77aac26bcd514"
server_key_iv: "7fe40f6a575727f5d9f6eb8c8181d929"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.415957 2611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "adde224930134115a5f3e6137cbb57e0"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "d494ac0d480a1c6913b77aac26bcd514"
server_key_iv: "7fe40f6a575727f5d9f6eb8c8181d929"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.419865 2611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:51.422379 2626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:51.423794 2611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:51.423892 2611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "adde224930134115a5f3e6137cbb57e0"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "d494ac0d480a1c6913b77aac26bcd514"
server_key_iv: "7fe40f6a575727f5d9f6eb8c8181d929"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.424002 2611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:51.442782 2611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:51.446058 2611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:51.446228 2611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:51.447703 2611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:51.447753 2611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:51.447791 2611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:51.447821 2611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:51.458027 2611 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:41867
I20260929 18:35:51.458082 2741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:41867 every 8 connection(s)
I20260929 18:35:51.459116 2611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:51.461877 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.448637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:33729 (local address 127.0.198.62:37275)
0929 18:35:51.448791 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.448796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.449736 (+ 940us) server_negotiation.cc:407] Connection header received
0929 18:35:51.450931 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.450935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.450998 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.451078 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.452726 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.453338 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.454763 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.455131 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.455249 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.458025 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.458029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.458032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.458062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.459618 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.460318 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.460321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.460323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.460402 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.460702 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.460705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.460707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.460939 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.461027 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.461630 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.461750 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:51.462606 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.449060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37275 (local address 127.0.198.1:33729)
0929 18:35:51.449572 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.449622 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.450714 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.451289 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.451298 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.451793 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:51.452542 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.452564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.453466 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.453470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.454641 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.454652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.455280 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.455283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.455441 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.456093 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:51.456116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:51.457803 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:51.459827 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.459834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.459849 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.460217 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.460485 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.460487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.460489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.460609 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.461050 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:51.461055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:51.461292 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.461446 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.461729 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260929 18:35:51.464071 2742 heartbeater.cc:347] Connected to a master server at 127.0.198.62:37275
I20260929 18:35:51.464234 2742 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:51.464746 2742 heartbeater.cc:511] Master 127.0.198.62:37275 requested a full tablet report, sending...
I20260929 18:35:51.466318 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 2611
I20260929 18:35:51.466301 2539 ts_manager.cc:194] Registered new tserver with Master: adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867)
I20260929 18:35:51.466435 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:51.466742 792 external_mini_cluster.cc:1428] Setting key febe862762203643399d50860c96ff3e
I20260929 18:35:51.467793 2539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:33729
I20260929 18:35:51.474284 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.466374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:32971 (local address 127.0.198.62:37275)
0929 18:35:51.466561 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.466565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.466868 (+ 303us) server_negotiation.cc:407] Connection header received
0929 18:35:51.467014 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.467017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.467060 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.467131 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.467613 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.468290 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.469423 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.469747 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.469810 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.471082 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.471086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.471089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.471115 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.472532 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.473058 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.473060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.473061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.473135 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.473406 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.473408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.473409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.473592 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.473678 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.474058 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.474165 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 18:35:51.474318 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.466318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37275 (local address 127.0.198.1:32971)
0929 18:35:51.466789 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.466803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.466929 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.467165 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.467168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.467381 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:51.467495 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.467502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.468449 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.468452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.469275 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.469286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.469948 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.469950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.470062 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.470581 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:51.470591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:51.470931 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:51.472663 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.472668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.472670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.472926 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.473219 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.473222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.473224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.473324 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.473693 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:51.473696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:51.473804 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.473916 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.474112 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":308,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:35:51.498031 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37275
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:51.610942 2751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:51.611239 2751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:51.611289 2751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:51.615430 2751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:51.615499 2751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:51.615546 2751 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:35:51.615559 2751 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:35:51.615590 2751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:35:51.620947 2751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:51.622027 2751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:51.623023 2751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:51.633072 2759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:51.633235 2751 server_base.cc:1034] running on GCE node
W20260929 18:35:51.633087 2757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:51.633072 2756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:51.633813 2751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:51.634521 2751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:51.635697 2751 hybrid_clock.cc:648] HybridClock initialized: now 1790706951635684 us; error 27 us; skew 500 ppm
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:51.638664 2751 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:35:51.639586 2751 webserver.cc:460] Webserver started at http://127.0.198.2:37225/ using document root <none> and password file <none>
I20260929 18:35:51.640132 2751 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:51.640192 2751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:51.640383 2751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:51.642524 2751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "136c1fbf4742488faad05863f6b0ccf5"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "7b1a985bdf23afe70c7896db5aefe08d"
server_key_iv: "901818074d819b00173ef06c8affd391"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.643066 2751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "136c1fbf4742488faad05863f6b0ccf5"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "7b1a985bdf23afe70c7896db5aefe08d"
server_key_iv: "901818074d819b00173ef06c8affd391"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.646966 2751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:35:51.649477 2766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:51.650942 2751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:51.651059 2751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "136c1fbf4742488faad05863f6b0ccf5"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "7b1a985bdf23afe70c7896db5aefe08d"
server_key_iv: "901818074d819b00173ef06c8affd391"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.651170 2751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:51.693827 2751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:51.697279 2751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:51.697479 2751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:51.699039 2751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:51.699090 2751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:51.699143 2751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:51.699172 2751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:51.709369 2751 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42117
I20260929 18:35:51.709420 2881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42117 every 8 connection(s)
I20260929 18:35:51.710348 2751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:51.712481 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.699946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:44139 (local address 127.0.198.62:37275)
0929 18:35:51.700065 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.700068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.701155 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 18:35:51.702288 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.702292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.702349 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.702426 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.703737 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.704391 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.706091 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.706321 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.706424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.708991 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.708995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.708997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.709024 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.710527 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.711078 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.711082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.711096 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.711186 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.711524 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.711526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.711527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.711703 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.711776 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.712260 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.712361 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:35:51.713270 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.700489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37275 (local address 127.0.198.2:44139)
0929 18:35:51.700982 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.701027 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.702066 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.702641 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.702651 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.703000 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:51.703595 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.703611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.704497 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.704500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.705923 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.705934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.706453 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.706457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.706658 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.707243 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:51.707261 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:51.708825 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:51.710630 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.710635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.710651 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.710964 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.711284 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.711287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.711289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.711439 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.711787 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:51.711793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:51.712045 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.712202 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.712479 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260929 18:35:51.714530 2882 heartbeater.cc:347] Connected to a master server at 127.0.198.62:37275
I20260929 18:35:51.714671 2882 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:51.714751 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 2751
I20260929 18:35:51.714854 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:51.715175 792 external_mini_cluster.cc:1428] Setting key 5130b271f50985cd2652bcf170c5caa7
I20260929 18:35:51.715212 2882 heartbeater.cc:511] Master 127.0.198.62:37275 requested a full tablet report, sending...
I20260929 18:35:51.716112 2539 ts_manager.cc:194] Registered new tserver with Master: 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117)
I20260929 18:35:51.716734 2539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:44139
I20260929 18:35:51.723945 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.716771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:60697 (local address 127.0.198.62:37275)
0929 18:35:51.716873 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.716877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.716965 (+ 88us) server_negotiation.cc:407] Connection header received
0929 18:35:51.717094 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.717097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.717140 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.717218 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.717714 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.718473 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.719575 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.719794 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.719878 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.721064 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.721068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.721070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.721092 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.722465 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.722848 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.722852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.722854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.722921 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.723161 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.723164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.723166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.723367 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.723451 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.723691 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.723791 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:35:51.723955 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.716642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37275 (local address 127.0.198.2:60697)
0929 18:35:51.716887 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.716901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.717004 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.717248 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.717251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.717467 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:51.717598 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.717605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.718606 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.718609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.719445 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.719456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.719901 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.719904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.720000 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.720496 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:51.720510 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:51.720931 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:51.722570 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.722574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.722576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.722765 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.722998 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.723001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.723003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.723085 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.723453 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:51.723456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:51.723544 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.723630 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.723772 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:35:51.742229 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37275
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:51.852450 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:51.852743 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:51.852811 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:51.856940 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:51.857005 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:51.857069 2891 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:35:51.857092 2891 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:35:51.857139 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:35:51.862871 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:51.863955 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:51.864967 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:51.874318 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:51.874421 2891 server_base.cc:1034] running on GCE node
W20260929 18:35:51.874264 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:51.874262 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:51.874943 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:51.875579 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:51.876745 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790706951876724 us; error 34 us; skew 500 ppm
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:51.879536 2891 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:35:51.880496 2891 webserver.cc:460] Webserver started at http://127.0.198.3:45337/ using document root <none> and password file <none>
I20260929 18:35:51.881039 2891 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:51.881099 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:51.881284 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:51.883436 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "db3fa85fcc314dffb21564b285ff4f84"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "b9f0ed78f818dad3f20f664de9d57362"
server_key_iv: "e36e2b525621ee00d34b6661fe613fd9"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.884013 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "db3fa85fcc314dffb21564b285ff4f84"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "b9f0ed78f818dad3f20f664de9d57362"
server_key_iv: "e36e2b525621ee00d34b6661fe613fd9"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.887897 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 18:35:51.890408 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:51.891650 2891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:51.891754 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "db3fa85fcc314dffb21564b285ff4f84"
format_stamp: "Formatted at 2026-09-29 18:35:51 on dist-test-slave-q5qx"
server_key: "b9f0ed78f818dad3f20f664de9d57362"
server_key_iv: "e36e2b525621ee00d34b6661fe613fd9"
server_key_version: "encryptionkey@0"
I20260929 18:35:51.891862 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:51.915949 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:51.919006 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:51.919178 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:51.920588 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:51.920643 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:51.920686 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:51.920712 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:51.930820 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:38037
I20260929 18:35:51.930878 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:38037 every 8 connection(s)
I20260929 18:35:51.931912 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:51.934377 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.921495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:40143 (local address 127.0.198.62:37275)
0929 18:35:51.921632 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.921637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.922563 (+ 926us) server_negotiation.cc:407] Connection header received
0929 18:35:51.923779 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.923782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.923842 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.923943 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.925655 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.926506 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.927887 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.928179 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.928248 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.930897 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.930901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.930904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.930931 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.932384 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.932972 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.932975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.932977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.933058 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.933359 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.933362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.933365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.933542 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.933631 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.934096 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.934218 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:51.935194 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.921876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37275 (local address 127.0.198.3:40143)
0929 18:35:51.922386 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.922446 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.923530 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.924266 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.924276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.924710 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:51.925435 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.925456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.926656 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.926660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.927741 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.927752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.928415 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.928418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.928582 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.929225 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:51.929242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:51.930725 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:51.932512 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.932517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.932534 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.932868 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.933142 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.933145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.933147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.933274 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.933632 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:51.933638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:51.933872 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.934057 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.934397 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260929 18:35:51.936560 3022 heartbeater.cc:347] Connected to a master server at 127.0.198.62:37275
I20260929 18:35:51.936717 3022 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:51.937243 3022 heartbeater.cc:511] Master 127.0.198.62:37275 requested a full tablet report, sending...
I20260929 18:35:51.938364 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 2891
I20260929 18:35:51.938390 2538 ts_manager.cc:194] Registered new tserver with Master: db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:51.938479 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:51.938809 792 external_mini_cluster.cc:1428] Setting key 93dac752d232f0f9d8254c67c3ff5948
I20260929 18:35:51.939126 2538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:40143
I20260929 18:35:51.940801 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:51.945807 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.938590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37275 (local address 127.0.198.3:33737)
0929 18:35:51.938953 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.938966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.939059 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.939308 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.939310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.939524 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:51.939839 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.939847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.940584 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.940587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.941209 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.941217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.941630 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.941634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.941716 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.942261 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:51.942273 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:51.942739 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:51.944370 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.944373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.944375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.944624 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.944866 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:51.944869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:51.944871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:51.944960 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:51.945311 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:51.945314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:51.945414 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.945527 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.945648 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":93,"threads_started":1}
I20260929 18:35:51.945873 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.938693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:33737 (local address 127.0.198.62:37275)
0929 18:35:51.938799 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.938803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.939039 (+ 236us) server_negotiation.cc:407] Connection header received
0929 18:35:51.939151 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.939154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.939205 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.939279 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.939936 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.940449 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.941324 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.941535 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.941616 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.942879 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.942882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.942884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.942906 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.944260 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.944711 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.944714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.944715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.944785 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.945047 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.945049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.945050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.945226 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.945299 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.945605 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.945722 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:35:51 dist-test-slave-q5qx krb5kdc[2494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:51.966763 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.958433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.0.198.62:37275)
0929 18:35:51.958569 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.958572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.958583 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:35:51.958623 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.958626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.958662 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.958732 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:51.959433 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.960012 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.960830 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.961041 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.961126 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.963645 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:51.963649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:51.963651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:51.963678 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:51.965280 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.965772 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.965775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.965776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.965855 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:51.966094 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:51.966097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:51.966098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:51.966270 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:51.966360 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.966544 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.966638 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:35:51.969666 2539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:51.972741 2539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:51.986708 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.981245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42117 (local address 127.0.0.1:46244)
0929 18:35:51.982204 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.982217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.982295 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.982880 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.982883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.982909 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:51.983057 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.983064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.984130 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.984134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.985342 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.985351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.986226 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.986229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.986315 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.986346 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.986461 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.986569 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":55,"spinlock_wait_cycles":896,"thread_start_us":104,"threads_started":1}
I20260929 18:35:51.986734 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.981790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.0.198.2:42117)
0929 18:35:51.982510 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.982521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.982539 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:35:51.982605 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.982609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.982742 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.983193 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:51.983308 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.984010 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.985460 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.986094 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.986202 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.986370 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.986492 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.986588 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":92,"threads_started":1}
I20260929 18:35:51.987562 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.981421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:38037 (local address 127.0.0.1:48184)
0929 18:35:51.982470 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.982482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.982572 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.983008 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.983010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.983024 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:51.983119 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.983124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.984497 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.984500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.985904 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.985914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.987175 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.987178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.987252 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.987271 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.987337 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.987416 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":66,"thread_start_us":93,"threads_started":1}
I20260929 18:35:51.987599 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.981876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.0.198.3:38037)
0929 18:35:51.982313 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.982324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.982574 (+ 250us) server_negotiation.cc:407] Connection header received
0929 18:35:51.982678 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.982684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.982851 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.982990 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:51.983206 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.984340 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.986056 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.986836 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.986938 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.987077 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.987389 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.987476 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":133,"threads_started":1}
I20260929 18:35:51.988610 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.981126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41867 (local address 127.0.0.1:33408)
0929 18:35:51.981894 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:51.981946 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:51.982101 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:51.982844 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:51.982851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:51.982873 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:51.983041 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.983050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.984472 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.984476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.985904 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:51.985914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.988220 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.988224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:51.988317 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.988339 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:35:51.988414 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:35:51.988480 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":158,"thread_start_us":61,"threads_started":1}
I20260929 18:35:51.988677 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:51.981420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.0.198.1:41867)
0929 18:35:51.982355 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:51.982361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:51.982384 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:35:51.982460 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:51.982465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:51.982697 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:51.982853 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:51.983149 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.984342 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.987138 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:51.987931 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:51.988033 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:51.988161 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:51.988473 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 18:35:51.988573 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":110,"threads_started":1}
I20260929 18:35:51.990374 2816 tablet_service.cc:1467] Processing CreateTablet for tablet 7091b8eca894404bbdbe78642c824e57 (DEFAULT_TABLE table=test-table [id=66325214149c448e91f1e63412c8dd67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:51.990706 2956 tablet_service.cc:1467] Processing CreateTablet for tablet 7091b8eca894404bbdbe78642c824e57 (DEFAULT_TABLE table=test-table [id=66325214149c448e91f1e63412c8dd67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:51.991633 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7091b8eca894404bbdbe78642c824e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:51.991662 2816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7091b8eca894404bbdbe78642c824e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:51.992305 2676 tablet_service.cc:1467] Processing CreateTablet for tablet 7091b8eca894404bbdbe78642c824e57 (DEFAULT_TABLE table=test-table [id=66325214149c448e91f1e63412c8dd67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:51.993147 2676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7091b8eca894404bbdbe78642c824e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:52.000130 3046 tablet_bootstrap.cc:492] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Bootstrap starting.
I20260929 18:35:52.000130 3047 tablet_bootstrap.cc:492] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Bootstrap starting.
I20260929 18:35:52.000150 3045 tablet_bootstrap.cc:492] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Bootstrap starting.
I20260929 18:35:52.002136 3045 tablet_bootstrap.cc:654] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.002249 3046 tablet_bootstrap.cc:654] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.002249 3047 tablet_bootstrap.cc:654] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.002890 3045 log.cc:824] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:52.003052 3046 log.cc:824] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:52.003051 3047 log.cc:824] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:52.004943 3045 tablet_bootstrap.cc:492] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: No bootstrap required, opened a new log
I20260929 18:35:52.004979 3047 tablet_bootstrap.cc:492] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: No bootstrap required, opened a new log
I20260929 18:35:52.005156 3047 ts_tablet_manager.cc:1397] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:35:52.005163 3045 ts_tablet_manager.cc:1397] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:35:52.004979 3046 tablet_bootstrap.cc:492] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: No bootstrap required, opened a new log
I20260929 18:35:52.005434 3046 ts_tablet_manager.cc:1397] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:35:52.008884 3045 raft_consensus.cc:348] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.009161 3045 raft_consensus.cc:374] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.009218 3045 raft_consensus.cc:729] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adde224930134115a5f3e6137cbb57e0, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.009274 3046 raft_consensus.cc:348] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.009274 3047 raft_consensus.cc:348] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.009578 3046 raft_consensus.cc:374] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.009578 3047 raft_consensus.cc:374] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.009636 3047 raft_consensus.cc:729] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3fa85fcc314dffb21564b285ff4f84, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.009636 3046 raft_consensus.cc:729] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136c1fbf4742488faad05863f6b0ccf5, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.009718 3045 consensus_queue.cc:260] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.010092 3047 consensus_queue.cc:260] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.010092 3046 consensus_queue.cc:260] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.010486 2742 heartbeater.cc:503] Master 127.0.198.62:37275 was elected leader, sending a full tablet report...
I20260929 18:35:52.010835 3045 ts_tablet_manager.cc:1428] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 18:35:52.011114 3047 ts_tablet_manager.cc:1428] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:35:52.011114 3046 ts_tablet_manager.cc:1428] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:35:52.011466 3022 heartbeater.cc:503] Master 127.0.198.62:37275 was elected leader, sending a full tablet report...
I20260929 18:35:52.012501 2882 heartbeater.cc:503] Master 127.0.198.62:37275 was elected leader, sending a full tablet report...
I20260929 18:35:52.084246 3051 raft_consensus.cc:482] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:52.084475 3051 raft_consensus.cc:504] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.085701 3051 leader_election.cc:288] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:52.089659 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.086126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:52721 (local address 127.0.198.2:42117)
0929 18:35:52.086276 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.086281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.086522 (+ 241us) server_negotiation.cc:407] Connection header received
0929 18:35:52.086650 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.086653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.086696 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.086755 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:52.087019 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.087524 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.088672 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.089091 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.089185 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.089246 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.089449 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.089520 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:35:52.089659 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.086211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:60503 (local address 127.0.198.3:38037)
0929 18:35:52.086312 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.086316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.086605 (+ 289us) server_negotiation.cc:407] Connection header received
0929 18:35:52.086737 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.086740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.086785 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.086842 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:52.087103 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.087601 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.088662 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.089080 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.089174 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.089237 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.089443 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.089527 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:35:52.089684 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.086012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42117 (local address 127.0.198.1:52721)
0929 18:35:52.086417 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.086430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.086547 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.086781 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.086783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.086798 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:52.086910 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.086915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.087640 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.087643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.088548 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.088557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.089208 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.089212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.089333 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.089350 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.089425 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.089520 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":15,"thread_start_us":139,"threads_started":1}
I20260929 18:35:52.089689 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.086012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:38037 (local address 127.0.198.1:60503)
0929 18:35:52.086509 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.086520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.086632 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.086864 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.086866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.086881 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:52.086988 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.086993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.087703 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.087706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.088549 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.088559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.089204 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.089208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.089329 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.089347 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.089426 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.089529 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":94,"thread_start_us":109,"threads_started":1}
I20260929 18:35:52.090348 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091b8eca894404bbdbe78642c824e57" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84" is_pre_election: true
I20260929 18:35:52.090350 2835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091b8eca894404bbdbe78642c824e57" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c1fbf4742488faad05863f6b0ccf5" is_pre_election: true
I20260929 18:35:52.090631 2976 raft_consensus.cc:2454] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 0.
I20260929 18:35:52.090647 2835 raft_consensus.cc:2454] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 0.
I20260929 18:35:52.091125 2630 leader_election.cc:302] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adde224930134115a5f3e6137cbb57e0, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:52.091382 3051 raft_consensus.cc:2781] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:52.091456 3051 raft_consensus.cc:482] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:52.091487 3051 raft_consensus.cc:3037] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.092521 3051 raft_consensus.cc:504] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.092885 3051 leader_election.cc:288] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 election: Requested vote from peers 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:52.093274 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091b8eca894404bbdbe78642c824e57" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84"
I20260929 18:35:52.093274 2835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091b8eca894404bbdbe78642c824e57" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c1fbf4742488faad05863f6b0ccf5"
I20260929 18:35:52.093410 2976 raft_consensus.cc:3037] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.093410 2835 raft_consensus.cc:3037] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.094774 2835 raft_consensus.cc:2454] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 1.
I20260929 18:35:52.094774 2976 raft_consensus.cc:2454] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 1.
I20260929 18:35:52.095124 2630 leader_election.cc:302] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 136c1fbf4742488faad05863f6b0ccf5, adde224930134115a5f3e6137cbb57e0, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:52.095304 3051 raft_consensus.cc:2781] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:52.095588 3051 raft_consensus.cc:686] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 1 LEADER]: Becoming Leader. State: Replica: adde224930134115a5f3e6137cbb57e0, State: Running, Role: LEADER
I20260929 18:35:52.095966 3051 consensus_queue.cc:237] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.099188 2539 catalog_manager.cc:5495] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to adde224930134115a5f3e6137cbb57e0 (127.0.198.1). New cstate: current_term: 1 leader_uuid: "adde224930134115a5f3e6137cbb57e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:35:52 dist-test-slave-q5qx krb5kdc[2494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:52.140296 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.131765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37275 (local address 127.0.0.1:59452)
0929 18:35:52.132127 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.132139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.132270 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.132503 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.132506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.133006 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:52.133121 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.133129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.133854 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.133857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.134540 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.134551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.134948 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.134951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.135071 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.135702 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:52.135715 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:52.137213 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:52.138802 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:52.138814 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:52.138821 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:52.139052 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:52.139283 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:52.139286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:52.139288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:52.139366 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:52.139708 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:52.139716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:52.139826 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.139920 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.140083 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":32,"thread_start_us":138,"threads_started":1}
I20260929 18:35:52.140296 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.131867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.198.62:37275)
0929 18:35:52.132020 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.132025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.132229 (+ 204us) server_negotiation.cc:407] Connection header received
0929 18:35:52.132370 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.132373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.132408 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.132473 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:52.133250 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.133754 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.134654 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.134853 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.134937 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.137349 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:52.137353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:52.137355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:52.137378 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:52.138704 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:52.139136 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:52.139139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:52.139141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:52.139207 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:52.139437 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:52.139440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:52.139442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:52.139617 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:52.139698 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.139983 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.140132 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":46}
I20260929 18:35:52.144845 2539 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:59452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:35:52.145648 2539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:52.150890 2816 tablet_service.cc:1467] Processing CreateTablet for tablet d6782b0c495842cf9736b85bf54c80d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04b305e41653488e8e57e0c2ec752250]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:35:52.150890 2676 tablet_service.cc:1467] Processing CreateTablet for tablet d6782b0c495842cf9736b85bf54c80d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04b305e41653488e8e57e0c2ec752250]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:35:52.151062 2956 tablet_service.cc:1467] Processing CreateTablet for tablet d6782b0c495842cf9736b85bf54c80d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04b305e41653488e8e57e0c2ec752250]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:35:52.151698 2816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6782b0c495842cf9736b85bf54c80d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:52.151698 2676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6782b0c495842cf9736b85bf54c80d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:52.151716 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6782b0c495842cf9736b85bf54c80d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:52.154160 3047 tablet_bootstrap.cc:492] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: Bootstrap starting.
I20260929 18:35:52.154675 3046 tablet_bootstrap.cc:492] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: Bootstrap starting.
I20260929 18:35:52.155265 3047 tablet_bootstrap.cc:654] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.155458 3045 tablet_bootstrap.cc:492] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: Bootstrap starting.
I20260929 18:35:52.155643 3046 tablet_bootstrap.cc:654] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.156446 3045 tablet_bootstrap.cc:654] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.156733 3047 tablet_bootstrap.cc:492] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: No bootstrap required, opened a new log
I20260929 18:35:52.156756 3046 tablet_bootstrap.cc:492] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: No bootstrap required, opened a new log
I20260929 18:35:52.156819 3046 ts_tablet_manager.cc:1397] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:52.156821 3047 ts_tablet_manager.cc:1397] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:52.157265 3047 raft_consensus.cc:348] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.157265 3046 raft_consensus.cc:348] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.157387 3047 raft_consensus.cc:374] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.157408 3046 raft_consensus.cc:374] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.157425 3047 raft_consensus.cc:729] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3fa85fcc314dffb21564b285ff4f84, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.157440 3046 raft_consensus.cc:729] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136c1fbf4742488faad05863f6b0ccf5, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.157552 3046 consensus_queue.cc:260] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.157580 3047 consensus_queue.cc:260] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.157807 3046 ts_tablet_manager.cc:1428] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:52.157807 3047 ts_tablet_manager.cc:1428] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:52.158201 3052 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.158352 3045 tablet_bootstrap.cc:492] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: No bootstrap required, opened a new log
I20260929 18:35:52.158350 3052 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:52.158421 3045 ts_tablet_manager.cc:1397] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:35:52.158890 3053 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.158893 3045 raft_consensus.cc:348] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.159019 3045 raft_consensus.cc:374] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.159039 3053 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:52.159050 3045 raft_consensus.cc:729] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adde224930134115a5f3e6137cbb57e0, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.159225 3045 consensus_queue.cc:260] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.159502 3051 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.159657 3051 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:52.159500 3045 ts_tablet_manager.cc:1428] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260929 18:35:52.182916 3023 tablet.cc:2369] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:52.210546 2743 tablet.cc:2369] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:52.211604 2883 tablet.cc:2369] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:52.297412 3052 raft_consensus.cc:482] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:52.297652 3052 raft_consensus.cc:504] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.298815 3052 leader_election.cc:288] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:52.303038 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.299206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:54511 (local address 127.0.198.3:38037)
0929 18:35:52.299330 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.299334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.299584 (+ 250us) server_negotiation.cc:407] Connection header received
0929 18:35:52.299660 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.299663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.299715 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.299812 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:52.300208 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.300803 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.301957 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.302401 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.302483 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.302542 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.302853 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.302930 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:52.303032 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.299216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:37359 (local address 127.0.198.1:41867)
0929 18:35:52.299349 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.299353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.299878 (+ 525us) server_negotiation.cc:407] Connection header received
0929 18:35:52.299998 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.300002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.300052 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.300130 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:52.300460 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.301043 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.302058 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.302466 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.302583 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.302644 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.302822 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.302906 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:35:52.303057 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.299123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:38037 (local address 127.0.198.2:54511)
0929 18:35:52.299470 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.299482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.299585 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.299846 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.299850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.299898 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:52.300029 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.300037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.300920 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.300923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.301821 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.301830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.302573 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.302577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.302735 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.302753 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.302824 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.302894 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":44,"spinlock_wait_cycles":25344,"thread_start_us":100,"threads_started":1}
I20260929 18:35:52.303057 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.299124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41867 (local address 127.0.198.2:37359)
0929 18:35:52.299778 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.299793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.299907 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.300205 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.300208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.300227 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:52.300324 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.300330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.301152 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.301155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.301927 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.301938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.302589 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.302592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.302711 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.302730 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.302789 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.302858 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":52,"spinlock_wait_cycles":3072,"thread_start_us":109,"threads_started":1}
I20260929 18:35:52.303550 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6782b0c495842cf9736b85bf54c80d2" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84" is_pre_election: true
I20260929 18:35:52.303704 2976 raft_consensus.cc:2454] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 0.
I20260929 18:35:52.303926 2696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6782b0c495842cf9736b85bf54c80d2" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adde224930134115a5f3e6137cbb57e0" is_pre_election: true
I20260929 18:35:52.304031 2770 leader_election.cc:302] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136c1fbf4742488faad05863f6b0ccf5, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:52.304139 2696 raft_consensus.cc:2454] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 0.
I20260929 18:35:52.304318 3052 raft_consensus.cc:2781] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:52.304379 3052 raft_consensus.cc:482] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:52.304426 3052 raft_consensus.cc:3037] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.305513 3052 raft_consensus.cc:504] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.305856 3052 leader_election.cc:288] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 election: Requested vote from peers adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:52.306249 2696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6782b0c495842cf9736b85bf54c80d2" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adde224930134115a5f3e6137cbb57e0"
I20260929 18:35:52.306272 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6782b0c495842cf9736b85bf54c80d2" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84"
I20260929 18:35:52.306372 2696 raft_consensus.cc:3037] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.306372 2976 raft_consensus.cc:3037] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.307441 2696 raft_consensus.cc:2454] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 1.
I20260929 18:35:52.307441 2976 raft_consensus.cc:2454] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 1.
I20260929 18:35:52.307693 2770 leader_election.cc:302] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136c1fbf4742488faad05863f6b0ccf5, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:52.307842 3052 raft_consensus.cc:2781] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:52.308013 3052 raft_consensus.cc:686] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [term 1 LEADER]: Becoming Leader. State: Replica: 136c1fbf4742488faad05863f6b0ccf5, State: Running, Role: LEADER
I20260929 18:35:52.308347 3052 consensus_queue.cc:237] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.310297 3069 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: TxnStatusTablet state changed. Reason: New leader 136c1fbf4742488faad05863f6b0ccf5. Latest consensus state: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.310397 3069 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:35:52.310753 3071 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:35:52.310971 3071 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:35:52.311234 2539 catalog_manager.cc:5495] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:52.354020 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.350543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.0.198.2:42117)
0929 18:35:52.350734 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.350738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.350753 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:35:52.350904 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.350907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.350961 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.351041 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:52.351276 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.351775 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.352595 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.352805 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.352889 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.353173 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.353618 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.353721 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.353794 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.353847 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260929 18:35:52.354020 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.350457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42117 (local address 127.0.0.1:46250)
0929 18:35:52.350613 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.350629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.350742 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.351059 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.351062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.351088 (+ 26us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:52.351184 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.351189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.351872 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.351875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.352480 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.352489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.352899 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.352903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.352989 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.353033 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.353706 (+ 673us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.353722 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.353784 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.353872 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:35:52.360319 2976 raft_consensus.cc:1261] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Refusing update from remote peer 136c1fbf4742488faad05863f6b0ccf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:52.360329 2696 raft_consensus.cc:1261] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Refusing update from remote peer 136c1fbf4742488faad05863f6b0ccf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:52.360957 3052 consensus_queue.cc:1035] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:52.361147 3069 consensus_queue.cc:1035] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:52.364449 3051 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: TxnStatusTablet state changed. Reason: New leader 136c1fbf4742488faad05863f6b0ccf5. Latest consensus state: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.364574 3051 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:52.364734 3053 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: TxnStatusTablet state changed. Reason: New leader 136c1fbf4742488faad05863f6b0ccf5. Latest consensus state: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.364836 3053 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:52.367380 3069 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.367498 3069 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:35:52.367597 3069 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.367666 3069 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P 136c1fbf4742488faad05863f6b0ccf5: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:35:52.368076 3053 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.368186 3053 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P db3fa85fcc314dffb21564b285ff4f84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:52.368865 3075 mvcc.cc:205] Tried to move back new op lower bound from 7334735676860108800 to 7334735676657520640. Current Snapshot: MvccSnapshot[applied={T|T < 7334735676860108800}]
I20260929 18:35:52.369012 3076 mvcc.cc:205] Tried to move back new op lower bound from 7334735676860108800 to 7334735676657520640. Current Snapshot: MvccSnapshot[applied={T|T < 7334735676860108800}]
I20260929 18:35:52.369553 3051 tablet_replica.cc:439] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } } }
I20260929 18:35:52.369663 3051 tablet_replica.cc:442] T d6782b0c495842cf9736b85bf54c80d2 P adde224930134115a5f3e6137cbb57e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:35:52.372709 3074 mvcc.cc:205] Tried to move back new op lower bound from 7334735676860108800 to 7334735676657520640. Current Snapshot: MvccSnapshot[applied={T|T < 7334735676860108800 or (T in {7334735676860108800})}]
I20260929 18:35:52.377630 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.374114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.0.198.1:41867)
0929 18:35:52.374218 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.374221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.374306 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:35:52.374452 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.374454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.374509 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.374598 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:52.374842 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.375427 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.376277 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.376497 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.376581 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.376840 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.377268 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.377366 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.377437 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.377501 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:35:52.389343 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.385613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:35557 (local address 127.0.198.2:42117)
0929 18:35:52.385748 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.385753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.385962 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:35:52.386127 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.386129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.386174 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.386226 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:52.386470 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.387064 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.388300 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.388554 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.388641 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.388965 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.389037 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.389081 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.389184 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.389224 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:35:52.389439 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.385574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42117 (local address 127.0.198.1:35557)
0929 18:35:52.385876 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.385891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.386032 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.386250 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.386253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.386261 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:52.386387 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.386393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.387165 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.387169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.388169 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.388180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.388674 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.388678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.388770 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.388822 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.389111 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.389116 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.389178 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.389254 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
I20260929 18:35:52.395944 2976 raft_consensus.cc:1261] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Refusing update from remote peer adde224930134115a5f3e6137cbb57e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:52.395968 2835 raft_consensus.cc:1261] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Refusing update from remote peer adde224930134115a5f3e6137cbb57e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:52.396354 3056 consensus_queue.cc:1035] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:52.396554 3051 consensus_queue.cc:1035] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:52.421110 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.417183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41867 (local address 127.0.198.2:56335)
0929 18:35:52.417564 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.417576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.417671 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.417890 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.417891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.417897 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:52.418033 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.418039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.419038 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.419042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.419839 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.419850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.420436 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.420439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.420513 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.420562 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.420841 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.420845 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.420885 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.420951 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260929 18:35:52.421104 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.417274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:56335 (local address 127.0.198.1:41867)
0929 18:35:52.417387 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.417390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.417648 (+ 258us) server_negotiation.cc:407] Connection header received
0929 18:35:52.417745 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.417748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.417803 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.417885 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:52.418126 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.418895 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.419945 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.420296 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.420392 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.420675 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.420763 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.420832 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.420896 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.420965 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:35:52.451475 2535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:52.451675 2535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:52.454545 2535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Sending DeleteTablet for 3 replicas of tablet 7091b8eca894404bbdbe78642c824e57
I20260929 18:35:52.455365 2676 tablet_service.cc:1514] Processing DeleteTablet for tablet 7091b8eca894404bbdbe78642c824e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:52 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33408
I20260929 18:35:52.455468 2956 tablet_service.cc:1514] Processing DeleteTablet for tablet 7091b8eca894404bbdbe78642c824e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:52 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:48184
I20260929 18:35:52.455883 3096 tablet_replica.cc:330] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: stopping tablet replica
I20260929 18:35:52.456274 3096 raft_consensus.cc:2229] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:52.456491 3096 raft_consensus.cc:2258] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:52.457077 3095 tablet_replica.cc:330] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: stopping tablet replica
I20260929 18:35:52.457042 2816 tablet_service.cc:1514] Processing DeleteTablet for tablet 7091b8eca894404bbdbe78642c824e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:52 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260929 18:35:52.457392 3095 raft_consensus.cc:2229] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:52.457414 2535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:35:52.457729 3095 raft_consensus.cc:2258] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:35:52.457867 2535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:52.459347 3095 ts_tablet_manager.cc:1905] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:52.463011 3095 ts_tablet_manager.cc:1918] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:52.463126 3095 log.cc:1196] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/7091b8eca894404bbdbe78642c824e57
I20260929 18:35:52.463425 3095 ts_tablet_manager.cc:1939] T 7091b8eca894404bbdbe78642c824e57 P adde224930134115a5f3e6137cbb57e0: Deleting consensus metadata
I20260929 18:35:52.464437 2956 tablet_service.cc:1467] Processing CreateTablet for tablet b123bdc9bdfe48a98e893245cc594f1e (DEFAULT_TABLE table=test-table [id=652fce0f85d646c4a950839b43403384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:52.464717 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b123bdc9bdfe48a98e893245cc594f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:52.465006 3096 ts_tablet_manager.cc:1905] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:52.467073 3047 tablet_bootstrap.cc:492] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: Bootstrap starting.
I20260929 18:35:52.467726 3096 ts_tablet_manager.cc:1918] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:52.467816 3096 log.cc:1196] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/7091b8eca894404bbdbe78642c824e57
I20260929 18:35:52.467926 3047 tablet_bootstrap.cc:654] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.468096 3096 ts_tablet_manager.cc:1939] T 7091b8eca894404bbdbe78642c824e57 P db3fa85fcc314dffb21564b285ff4f84: Deleting consensus metadata
I20260929 18:35:52.469079 2524 catalog_manager.cc:4841] TS adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867): tablet 7091b8eca894404bbdbe78642c824e57 (table test-table [id=66325214149c448e91f1e63412c8dd67]) successfully deleted
I20260929 18:35:52.469460 2526 catalog_manager.cc:4841] TS db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037): tablet 7091b8eca894404bbdbe78642c824e57 (table test-table [id=66325214149c448e91f1e63412c8dd67]) successfully deleted
I20260929 18:35:52.470059 2676 tablet_service.cc:1467] Processing CreateTablet for tablet b123bdc9bdfe48a98e893245cc594f1e (DEFAULT_TABLE table=test-table [id=652fce0f85d646c4a950839b43403384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:52.470342 2676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b123bdc9bdfe48a98e893245cc594f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:52.470500 2816 tablet_service.cc:1467] Processing CreateTablet for tablet b123bdc9bdfe48a98e893245cc594f1e (DEFAULT_TABLE table=test-table [id=652fce0f85d646c4a950839b43403384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:52.470932 2816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b123bdc9bdfe48a98e893245cc594f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:52.472990 3045 tablet_bootstrap.cc:492] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: Bootstrap starting.
I20260929 18:35:52.473639 3046 tablet_bootstrap.cc:492] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: Bootstrap starting.
I20260929 18:35:52.473935 3045 tablet_bootstrap.cc:654] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.474481 3046 tablet_bootstrap.cc:654] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:52.475340 3045 tablet_bootstrap.cc:492] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: No bootstrap required, opened a new log
I20260929 18:35:52.475430 3045 ts_tablet_manager.cc:1397] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:35:52.475699 3097 tablet_replica.cc:330] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: stopping tablet replica
I20260929 18:35:52.475925 3047 tablet_bootstrap.cc:492] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: No bootstrap required, opened a new log
I20260929 18:35:52.475939 3045 raft_consensus.cc:348] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.476023 3047 ts_tablet_manager.cc:1397] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 18:35:52.476046 3045 raft_consensus.cc:374] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.476075 3045 raft_consensus.cc:729] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adde224930134115a5f3e6137cbb57e0, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.476171 3097 raft_consensus.cc:2229] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:52.476210 3045 consensus_queue.cc:260] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.476342 3097 raft_consensus.cc:2258] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:52.476459 3045 ts_tablet_manager.cc:1428] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 18:35:52.476471 3047 raft_consensus.cc:348] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.476568 3047 raft_consensus.cc:374] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.476598 3047 raft_consensus.cc:729] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3fa85fcc314dffb21564b285ff4f84, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.476719 3047 consensus_queue.cc:260] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.476962 3047 ts_tablet_manager.cc:1428] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:52.479156 3046 tablet_bootstrap.cc:492] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: No bootstrap required, opened a new log
I20260929 18:35:52.479259 3046 ts_tablet_manager.cc:1397] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 18:35:52.479681 3046 raft_consensus.cc:348] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.479794 3046 raft_consensus.cc:374] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:52.479820 3046 raft_consensus.cc:729] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136c1fbf4742488faad05863f6b0ccf5, State: Initialized, Role: FOLLOWER
I20260929 18:35:52.479930 3046 consensus_queue.cc:260] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.480175 3046 ts_tablet_manager.cc:1428] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:52.481863 3097 ts_tablet_manager.cc:1905] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:52.485116 3097 ts_tablet_manager.cc:1918] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:52.485224 3097 log.cc:1196] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/7091b8eca894404bbdbe78642c824e57
I20260929 18:35:52.485489 3097 ts_tablet_manager.cc:1939] T 7091b8eca894404bbdbe78642c824e57 P 136c1fbf4742488faad05863f6b0ccf5: Deleting consensus metadata
I20260929 18:35:52.486742 2525 catalog_manager.cc:4841] TS 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117): tablet 7091b8eca894404bbdbe78642c824e57 (table test-table [id=66325214149c448e91f1e63412c8dd67]) successfully deleted
I20260929 18:35:52.595959 3089 raft_consensus.cc:482] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:52.596109 3089 raft_consensus.cc:504] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.596542 3089 leader_election.cc:288] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:52.596999 2696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b123bdc9bdfe48a98e893245cc594f1e" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adde224930134115a5f3e6137cbb57e0" is_pre_election: true
I20260929 18:35:52.597028 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b123bdc9bdfe48a98e893245cc594f1e" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84" is_pre_election: true
I20260929 18:35:52.597146 2696 raft_consensus.cc:2454] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 0.
I20260929 18:35:52.597218 2976 raft_consensus.cc:2454] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 0.
I20260929 18:35:52.597553 2770 leader_election.cc:302] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 136c1fbf4742488faad05863f6b0ccf5, adde224930134115a5f3e6137cbb57e0, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:52.597775 3089 raft_consensus.cc:2781] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:52.597824 3089 raft_consensus.cc:482] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:52.597863 3089 raft_consensus.cc:3037] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.598872 3089 raft_consensus.cc:504] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.599218 3089 leader_election.cc:288] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 election: Requested vote from peers adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:52.599602 2696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b123bdc9bdfe48a98e893245cc594f1e" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adde224930134115a5f3e6137cbb57e0"
I20260929 18:35:52.599737 2696 raft_consensus.cc:3037] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.600862 2696 raft_consensus.cc:2454] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 1.
I20260929 18:35:52.601191 2768 leader_election.cc:302] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136c1fbf4742488faad05863f6b0ccf5, adde224930134115a5f3e6137cbb57e0; no voters:
I20260929 18:35:52.601161 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b123bdc9bdfe48a98e893245cc594f1e" candidate_uuid: "136c1fbf4742488faad05863f6b0ccf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84"
I20260929 18:35:52.601290 2976 raft_consensus.cc:3037] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:52.601362 3089 raft_consensus.cc:2781] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:52.601439 3089 raft_consensus.cc:686] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 1 LEADER]: Becoming Leader. State: Replica: 136c1fbf4742488faad05863f6b0ccf5, State: Running, Role: LEADER
I20260929 18:35:52.601601 3089 consensus_queue.cc:237] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:52.602362 2976 raft_consensus.cc:2454] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136c1fbf4742488faad05863f6b0ccf5 in term 1.
I20260929 18:35:52.603612 2535 catalog_manager.cc:5495] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "136c1fbf4742488faad05863f6b0ccf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:52.646011 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.643076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.0.198.2:42117)
0929 18:35:52.643190 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.643194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.643207 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:35:52.643308 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.643311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.643358 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.643429 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:52.643643 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.644209 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.645013 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.645241 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.645326 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.645559 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.645635 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.645690 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.645784 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.645833 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:35:52.654647 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.651790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:54023 (local address 127.0.198.2:42117)
0929 18:35:52.651871 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:52.651875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:52.651931 (+ 56us) server_negotiation.cc:407] Connection header received
0929 18:35:52.652086 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:52.652089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:52.652125 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:52.652182 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:52.652389 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.652919 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.653700 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.653891 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.653997 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.654230 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.654294 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.654350 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:52.654426 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:35:52.654496 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":30}
I20260929 18:35:52.654647 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:52.651732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42117 (local address 127.0.198.2:54023)
0929 18:35:52.651872 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:52.651887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:52.651998 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:52.652196 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:52.652198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:52.652204 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:52.652300 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.652306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.653024 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.653027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.653604 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:52.653611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:52.654008 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:52.654011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:52.654115 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:52.654140 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:52.654359 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:52.654364 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:35:52.654419 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:35:52.654527 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":37}
I20260929 18:35:52.660208 2976 raft_consensus.cc:1261] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Refusing update from remote peer 136c1fbf4742488faad05863f6b0ccf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:52.660504 2696 raft_consensus.cc:1261] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Refusing update from remote peer 136c1fbf4742488faad05863f6b0ccf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:52.660610 3052 consensus_queue.cc:1035] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:52.660832 3069 consensus_queue.cc:1035] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:35:52.673813 3074 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:35:54.228024 2535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:54.228183 2535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:54.231055 2535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Sending DeleteTablet for 3 replicas of tablet b123bdc9bdfe48a98e893245cc594f1e
I20260929 18:35:54.231699 2956 tablet_service.cc:1514] Processing DeleteTablet for tablet b123bdc9bdfe48a98e893245cc594f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:48184
I20260929 18:35:54.231699 2676 tablet_service.cc:1514] Processing DeleteTablet for tablet b123bdc9bdfe48a98e893245cc594f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33408
I20260929 18:35:54.231768 2816 tablet_service.cc:1514] Processing DeleteTablet for tablet b123bdc9bdfe48a98e893245cc594f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260929 18:35:54.232184 3125 tablet_replica.cc:330] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: stopping tablet replica
I20260929 18:35:54.232254 3126 tablet_replica.cc:330] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: stopping tablet replica
I20260929 18:35:54.232317 3125 raft_consensus.cc:2229] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:54.232353 3126 raft_consensus.cc:2229] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:54.232443 3125 raft_consensus.cc:2258] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:54.232461 3126 raft_consensus.cc:2258] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:54.233147 3126 ts_tablet_manager.cc:1905] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:54.233438 3125 ts_tablet_manager.cc:1905] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:54.233464 3127 tablet_replica.cc:330] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: stopping tablet replica
I20260929 18:35:54.233668 3127 raft_consensus.cc:2229] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:54.234090 3127 raft_consensus.cc:2258] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:54.234793 3127 ts_tablet_manager.cc:1905] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:54.236033 3125 ts_tablet_manager.cc:1918] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:35:54.236089 3126 ts_tablet_manager.cc:1918] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:35:54.236150 3126 log.cc:1196] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/b123bdc9bdfe48a98e893245cc594f1e
I20260929 18:35:54.236146 3125 log.cc:1196] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/b123bdc9bdfe48a98e893245cc594f1e
I20260929 18:35:54.236398 3126 ts_tablet_manager.cc:1939] T b123bdc9bdfe48a98e893245cc594f1e P adde224930134115a5f3e6137cbb57e0: Deleting consensus metadata
I20260929 18:35:54.236406 3125 ts_tablet_manager.cc:1939] T b123bdc9bdfe48a98e893245cc594f1e P db3fa85fcc314dffb21564b285ff4f84: Deleting consensus metadata
I20260929 18:35:54.237053 3127 ts_tablet_manager.cc:1918] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:35:54.237121 3127 log.cc:1196] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/b123bdc9bdfe48a98e893245cc594f1e
I20260929 18:35:54.237136 2526 catalog_manager.cc:4841] TS db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037): tablet b123bdc9bdfe48a98e893245cc594f1e (table test-table [id=652fce0f85d646c4a950839b43403384]) successfully deleted
I20260929 18:35:54.237285 2524 catalog_manager.cc:4841] TS adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867): tablet b123bdc9bdfe48a98e893245cc594f1e (table test-table [id=652fce0f85d646c4a950839b43403384]) successfully deleted
I20260929 18:35:54.237380 3127 ts_tablet_manager.cc:1939] T b123bdc9bdfe48a98e893245cc594f1e P 136c1fbf4742488faad05863f6b0ccf5: Deleting consensus metadata
I20260929 18:35:54.238072 2525 catalog_manager.cc:4841] TS 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117): tablet b123bdc9bdfe48a98e893245cc594f1e (table test-table [id=652fce0f85d646c4a950839b43403384]) successfully deleted
Sep 29 18:35:54 dist-test-slave-q5qx krb5kdc[2494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706954, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:35:54 dist-test-slave-q5qx krb5kdc[2494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706954, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:54.260447 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.252362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.0.198.62:37275)
0929 18:35:54.252632 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:54.252641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:54.252657 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:35:54.252752 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:54.252755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:54.252799 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:54.252877 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:54.253562 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.254158 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.254993 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.255215 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.255307 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.257549 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:54.257552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:54.257554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:54.257578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:54.258962 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:54.259325 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:54.259329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:54.259330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:54.259406 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:54.259651 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:54.259654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:54.259656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:54.259832 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:54.259946 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:54.260160 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:35:54.260284 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
I20260929 18:35:54.263046 2535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:54.263463 2535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:54.268065 2816 tablet_service.cc:1467] Processing CreateTablet for tablet c09b9658bc3a4271b629c3a7afa3028c (DEFAULT_TABLE table=test-table [id=079f2c7287ea43b3bf7a04452a802935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:54.268052 2676 tablet_service.cc:1467] Processing CreateTablet for tablet c09b9658bc3a4271b629c3a7afa3028c (DEFAULT_TABLE table=test-table [id=079f2c7287ea43b3bf7a04452a802935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:54.268177 2956 tablet_service.cc:1467] Processing CreateTablet for tablet c09b9658bc3a4271b629c3a7afa3028c (DEFAULT_TABLE table=test-table [id=079f2c7287ea43b3bf7a04452a802935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:54.268347 2676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09b9658bc3a4271b629c3a7afa3028c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:54.268347 2816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09b9658bc3a4271b629c3a7afa3028c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:54.268520 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09b9658bc3a4271b629c3a7afa3028c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:54.270931 3137 tablet_bootstrap.cc:492] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: Bootstrap starting.
I20260929 18:35:54.271095 3138 tablet_bootstrap.cc:492] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: Bootstrap starting.
I20260929 18:35:54.271481 3136 tablet_bootstrap.cc:492] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: Bootstrap starting.
I20260929 18:35:54.271785 3137 tablet_bootstrap.cc:654] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:54.271939 3138 tablet_bootstrap.cc:654] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:54.272351 3136 tablet_bootstrap.cc:654] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:54.273084 3137 tablet_bootstrap.cc:492] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: No bootstrap required, opened a new log
I20260929 18:35:54.273185 3137 ts_tablet_manager.cc:1397] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:54.273561 3137 raft_consensus.cc:348] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.273684 3137 raft_consensus.cc:374] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:54.273725 3137 raft_consensus.cc:729] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adde224930134115a5f3e6137cbb57e0, State: Initialized, Role: FOLLOWER
I20260929 18:35:54.273855 3137 consensus_queue.cc:260] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.274247 3137 ts_tablet_manager.cc:1428] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:54.274471 3136 tablet_bootstrap.cc:492] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: No bootstrap required, opened a new log
I20260929 18:35:54.274556 3136 ts_tablet_manager.cc:1397] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:54.274745 3138 tablet_bootstrap.cc:492] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: No bootstrap required, opened a new log
I20260929 18:35:54.274832 3138 ts_tablet_manager.cc:1397] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:35:54.275005 3136 raft_consensus.cc:348] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.275106 3136 raft_consensus.cc:374] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:54.275135 3136 raft_consensus.cc:729] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3fa85fcc314dffb21564b285ff4f84, State: Initialized, Role: FOLLOWER
I20260929 18:35:54.275216 3138 raft_consensus.cc:348] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.275314 3138 raft_consensus.cc:374] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:54.275282 3136 consensus_queue.cc:260] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.275354 3138 raft_consensus.cc:729] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136c1fbf4742488faad05863f6b0ccf5, State: Initialized, Role: FOLLOWER
I20260929 18:35:54.275472 3138 consensus_queue.cc:260] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.275641 3136 ts_tablet_manager.cc:1428] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:54.275705 3138 ts_tablet_manager.cc:1428] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:35:54.308877 3143 raft_consensus.cc:482] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:54.309010 3143 raft_consensus.cc:504] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.310216 3143 leader_election.cc:288] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867), 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117)
I20260929 18:35:54.313933 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.310540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:49061 (local address 127.0.198.2:42117)
0929 18:35:54.310841 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:54.310849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:54.310963 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:35:54.311087 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:54.311089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:54.311130 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:54.311199 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:54.311449 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.311954 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.312971 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.313396 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.313487 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.313554 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:54.313725 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:35:54.313799 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":83,"threads_started":1}
I20260929 18:35:54.313949 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.310490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42117 (local address 127.0.198.3:49061)
0929 18:35:54.310847 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:54.310860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:54.310991 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:54.311220 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:54.311222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:54.311239 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:54.311344 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.311349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.312074 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.312078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.312839 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.312849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.313512 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.313515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.313612 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.313633 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:35:54.313696 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:35:54.313771 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20260929 18:35:54.314086 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.310489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:56075 (local address 127.0.198.1:41867)
0929 18:35:54.310748 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:54.310757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:54.310839 (+ 82us) server_negotiation.cc:407] Connection header received
0929 18:35:54.310987 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:54.310990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:54.311031 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:54.311099 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:54.311356 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.311876 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.313034 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.313494 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.313593 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.313657 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:54.313832 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:35:54.313909 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260929 18:35:54.314085 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.310391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41867 (local address 127.0.198.3:56075)
0929 18:35:54.310746 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:54.310761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:54.310891 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:54.311123 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:54.311126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:54.311142 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:54.311257 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.311263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.311986 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.311990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.312816 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.312826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.313630 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.313633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.313727 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.313744 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:35:54.313808 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:35:54.313878 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260929 18:35:54.314579 2696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09b9658bc3a4271b629c3a7afa3028c" candidate_uuid: "db3fa85fcc314dffb21564b285ff4f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adde224930134115a5f3e6137cbb57e0" is_pre_election: true
I20260929 18:35:54.314606 2835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09b9658bc3a4271b629c3a7afa3028c" candidate_uuid: "db3fa85fcc314dffb21564b285ff4f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c1fbf4742488faad05863f6b0ccf5" is_pre_election: true
I20260929 18:35:54.314731 2696 raft_consensus.cc:2454] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3fa85fcc314dffb21564b285ff4f84 in term 0.
I20260929 18:35:54.314736 2835 raft_consensus.cc:2454] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3fa85fcc314dffb21564b285ff4f84 in term 0.
I20260929 18:35:54.315109 2908 leader_election.cc:302] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adde224930134115a5f3e6137cbb57e0, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:54.315380 3143 raft_consensus.cc:2781] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:54.315433 3143 raft_consensus.cc:482] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:54.315472 3143 raft_consensus.cc:3037] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:54.316459 3143 raft_consensus.cc:504] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.316797 3143 leader_election.cc:288] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [CANDIDATE]: Term 1 election: Requested vote from peers adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867), 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117)
I20260929 18:35:54.317140 2835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09b9658bc3a4271b629c3a7afa3028c" candidate_uuid: "db3fa85fcc314dffb21564b285ff4f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c1fbf4742488faad05863f6b0ccf5"
I20260929 18:35:54.317263 2696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09b9658bc3a4271b629c3a7afa3028c" candidate_uuid: "db3fa85fcc314dffb21564b285ff4f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adde224930134115a5f3e6137cbb57e0"
I20260929 18:35:54.317317 2835 raft_consensus.cc:3037] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:54.317382 2696 raft_consensus.cc:3037] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:54.318346 2696 raft_consensus.cc:2454] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3fa85fcc314dffb21564b285ff4f84 in term 1.
I20260929 18:35:54.318373 2835 raft_consensus.cc:2454] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3fa85fcc314dffb21564b285ff4f84 in term 1.
I20260929 18:35:54.318605 2908 leader_election.cc:302] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adde224930134115a5f3e6137cbb57e0, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:54.318771 3143 raft_consensus.cc:2781] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:54.318981 3143 raft_consensus.cc:686] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 1 LEADER]: Becoming Leader. State: Replica: db3fa85fcc314dffb21564b285ff4f84, State: Running, Role: LEADER
I20260929 18:35:54.319207 3143 consensus_queue.cc:237] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.321288 2539 catalog_manager.cc:5495] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3). New cstate: current_term: 1 leader_uuid: "db3fa85fcc314dffb21564b285ff4f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } health_report { overall_health: HEALTHY } } }
I20260929 18:35:54.344143 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.340501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.0.198.3:38037)
0929 18:35:54.340758 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:54.340765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:54.340776 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:35:54.340848 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:54.340850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:54.340900 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:54.341012 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:54.341265 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.341830 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.342708 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.342948 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.343028 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.343309 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:54.343765 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:54.343855 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:54.343953 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:35:54.344002 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260929 18:35:54.354457 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.351410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:43787 (local address 127.0.198.2:42117)
0929 18:35:54.351541 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:54.351545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:54.351644 (+ 99us) server_negotiation.cc:407] Connection header received
0929 18:35:54.351719 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:54.351721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:54.351768 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:54.351836 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:54.352089 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.352625 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.353513 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.353728 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.353828 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.354111 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:54.354181 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:54.354232 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:54.354292 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:35:54.354339 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:35:54.354535 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.351217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42117 (local address 127.0.198.3:43787)
0929 18:35:54.351546 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:54.351558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:54.351649 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:54.351846 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:54.351848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:54.351854 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:54.351970 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.351974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.352751 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.352753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.353389 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.353398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.353825 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.353828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.353923 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.353993 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:54.354240 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:54.354244 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:35:54.354286 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 18:35:54.354350 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":133,"threads_started":1}
I20260929 18:35:54.360002 2696 raft_consensus.cc:1261] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Refusing update from remote peer db3fa85fcc314dffb21564b285ff4f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:54.360110 2835 raft_consensus.cc:1261] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Refusing update from remote peer db3fa85fcc314dffb21564b285ff4f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:54.360440 3143 consensus_queue.cc:1035] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:54.360625 3148 consensus_queue.cc:1035] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:54.375306 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.372146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:41407 (local address 127.0.198.3:38037)
0929 18:35:54.372261 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:54.372265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:54.372477 (+ 212us) server_negotiation.cc:407] Connection header received
0929 18:35:54.372591 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:54.372594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:54.372638 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:54.372694 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:54.372912 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.373409 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.374289 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.374561 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.374647 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.374912 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:54.375001 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:54.375068 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:54.375156 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:35:54.375205 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:35:54.375339 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.372098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:38037 (local address 127.0.198.2:41407)
0929 18:35:54.372402 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:54.372414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:54.372506 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:54.372713 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:54.372716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:54.372725 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:35:54.372823 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.372829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.373510 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.373513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.374188 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:54.374197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.374661 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.374664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:54.374749 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.374779 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:35:54.375084 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:35:54.375092 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:35:54.375150 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:35:54.375230 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260929 18:35:54.397343 2539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:54.397495 2539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:54.399587 2539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Sending DeleteTablet for 3 replicas of tablet c09b9658bc3a4271b629c3a7afa3028c
I20260929 18:35:54.400218 2956 tablet_service.cc:1514] Processing DeleteTablet for tablet c09b9658bc3a4271b629c3a7afa3028c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:48184
I20260929 18:35:54.400238 2676 tablet_service.cc:1514] Processing DeleteTablet for tablet c09b9658bc3a4271b629c3a7afa3028c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33408
I20260929 18:35:54.400403 3125 tablet_replica.cc:330] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: stopping tablet replica
I20260929 18:35:54.400462 3126 tablet_replica.cc:330] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: stopping tablet replica
I20260929 18:35:54.400511 3125 raft_consensus.cc:2229] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:54.400565 3126 raft_consensus.cc:2229] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:54.400667 3126 raft_consensus.cc:2258] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:54.400837 3125 raft_consensus.cc:2258] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:54.401443 3126 ts_tablet_manager.cc:1905] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:54.401571 3125 ts_tablet_manager.cc:1905] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:54.401631 2816 tablet_service.cc:1514] Processing DeleteTablet for tablet c09b9658bc3a4271b629c3a7afa3028c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260929 18:35:54.401795 3127 tablet_replica.cc:330] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: stopping tablet replica
I20260929 18:35:54.401892 3127 raft_consensus.cc:2229] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:54.402027 3127 raft_consensus.cc:2258] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:54.402272 2539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:54.402676 2539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:54.404191 3126 ts_tablet_manager.cc:1918] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:54.404259 3126 log.cc:1196] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/c09b9658bc3a4271b629c3a7afa3028c
I20260929 18:35:54.404577 3126 ts_tablet_manager.cc:1939] T c09b9658bc3a4271b629c3a7afa3028c P adde224930134115a5f3e6137cbb57e0: Deleting consensus metadata
I20260929 18:35:54.405200 3127 ts_tablet_manager.cc:1905] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:54.405349 2524 catalog_manager.cc:4841] TS adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867): tablet c09b9658bc3a4271b629c3a7afa3028c (table test-table [id=079f2c7287ea43b3bf7a04452a802935]) successfully deleted
I20260929 18:35:54.408133 3127 ts_tablet_manager.cc:1918] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:54.408209 3127 log.cc:1196] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/c09b9658bc3a4271b629c3a7afa3028c
I20260929 18:35:54.408864 3125 ts_tablet_manager.cc:1918] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:35:54.409005 3125 log.cc:1196] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/c09b9658bc3a4271b629c3a7afa3028c
I20260929 18:35:54.409163 2956 tablet_service.cc:1467] Processing CreateTablet for tablet 8659acf4cb49495fbfa38fd2f01ae42a (DEFAULT_TABLE table=test-table [id=1f597b8c8cb842df9f945e0f8c564f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:54.409171 2676 tablet_service.cc:1467] Processing CreateTablet for tablet 8659acf4cb49495fbfa38fd2f01ae42a (DEFAULT_TABLE table=test-table [id=1f597b8c8cb842df9f945e0f8c564f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:54.409446 2676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8659acf4cb49495fbfa38fd2f01ae42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:54.409444 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8659acf4cb49495fbfa38fd2f01ae42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:54.409699 3127 ts_tablet_manager.cc:1939] T c09b9658bc3a4271b629c3a7afa3028c P 136c1fbf4742488faad05863f6b0ccf5: Deleting consensus metadata
I20260929 18:35:54.411598 3137 tablet_bootstrap.cc:492] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0: Bootstrap starting.
I20260929 18:35:54.411970 2525 catalog_manager.cc:4841] TS 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117): tablet c09b9658bc3a4271b629c3a7afa3028c (table test-table [id=079f2c7287ea43b3bf7a04452a802935]) successfully deleted
I20260929 18:35:54.412420 3137 tablet_bootstrap.cc:654] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:54.412549 2816 tablet_service.cc:1467] Processing CreateTablet for tablet 8659acf4cb49495fbfa38fd2f01ae42a (DEFAULT_TABLE table=test-table [id=1f597b8c8cb842df9f945e0f8c564f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:54.412820 2816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8659acf4cb49495fbfa38fd2f01ae42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:54.413435 3137 tablet_bootstrap.cc:492] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0: No bootstrap required, opened a new log
I20260929 18:35:54.413512 3137 ts_tablet_manager.cc:1397] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:54.413864 3137 raft_consensus.cc:348] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.413982 3137 raft_consensus.cc:374] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:54.414016 3137 raft_consensus.cc:729] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adde224930134115a5f3e6137cbb57e0, State: Initialized, Role: FOLLOWER
I20260929 18:35:54.414135 3137 consensus_queue.cc:260] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.414392 3137 ts_tablet_manager.cc:1428] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:54.415280 3138 tablet_bootstrap.cc:492] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: Bootstrap starting.
I20260929 18:35:54.416090 3138 tablet_bootstrap.cc:654] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:54.417220 3138 tablet_bootstrap.cc:492] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: No bootstrap required, opened a new log
I20260929 18:35:54.417307 3138 ts_tablet_manager.cc:1397] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:54.417665 3138 raft_consensus.cc:348] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.417786 3138 raft_consensus.cc:374] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:54.417829 3138 raft_consensus.cc:729] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136c1fbf4742488faad05863f6b0ccf5, State: Initialized, Role: FOLLOWER
I20260929 18:35:54.417964 3138 consensus_queue.cc:260] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.418253 3138 ts_tablet_manager.cc:1428] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:54.409442 3125 ts_tablet_manager.cc:1939] T c09b9658bc3a4271b629c3a7afa3028c P db3fa85fcc314dffb21564b285ff4f84: Deleting consensus metadata
I20260929 18:35:54.418848 3136 tablet_bootstrap.cc:492] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: Bootstrap starting.
I20260929 18:35:54.419612 3136 tablet_bootstrap.cc:654] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:54.420363 2526 catalog_manager.cc:4841] TS db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037): tablet c09b9658bc3a4271b629c3a7afa3028c (table test-table [id=079f2c7287ea43b3bf7a04452a802935]) successfully deleted
I20260929 18:35:54.421787 3136 tablet_bootstrap.cc:492] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: No bootstrap required, opened a new log
I20260929 18:35:54.422051 3136 ts_tablet_manager.cc:1397] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:54.422590 3136 raft_consensus.cc:348] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.422696 3136 raft_consensus.cc:374] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:54.422725 3136 raft_consensus.cc:729] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3fa85fcc314dffb21564b285ff4f84, State: Initialized, Role: FOLLOWER
I20260929 18:35:54.422868 3136 consensus_queue.cc:260] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.423147 3136 ts_tablet_manager.cc:1428] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:54.479094 3142 raft_consensus.cc:482] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:54.479246 3142 raft_consensus.cc:504] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.479661 3142 leader_election.cc:288] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:54.480113 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8659acf4cb49495fbfa38fd2f01ae42a" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84" is_pre_election: true
I20260929 18:35:54.480106 2835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8659acf4cb49495fbfa38fd2f01ae42a" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c1fbf4742488faad05863f6b0ccf5" is_pre_election: true
I20260929 18:35:54.480258 2976 raft_consensus.cc:2454] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 0.
I20260929 18:35:54.480258 2835 raft_consensus.cc:2454] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 0.
I20260929 18:35:54.480569 2630 leader_election.cc:302] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adde224930134115a5f3e6137cbb57e0, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:54.480759 3142 raft_consensus.cc:2781] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:54.480811 3142 raft_consensus.cc:482] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:54.480849 3142 raft_consensus.cc:3037] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:54.481814 3142 raft_consensus.cc:504] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.482182 3142 leader_election.cc:288] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 election: Requested vote from peers 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117), db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037)
I20260929 18:35:54.482532 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8659acf4cb49495fbfa38fd2f01ae42a" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fa85fcc314dffb21564b285ff4f84"
I20260929 18:35:54.482532 2835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8659acf4cb49495fbfa38fd2f01ae42a" candidate_uuid: "adde224930134115a5f3e6137cbb57e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136c1fbf4742488faad05863f6b0ccf5"
I20260929 18:35:54.482658 2835 raft_consensus.cc:3037] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:54.482658 2976 raft_consensus.cc:3037] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:54.483752 2976 raft_consensus.cc:2454] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 1.
I20260929 18:35:54.483745 2835 raft_consensus.cc:2454] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adde224930134115a5f3e6137cbb57e0 in term 1.
I20260929 18:35:54.484048 2630 leader_election.cc:302] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adde224930134115a5f3e6137cbb57e0, db3fa85fcc314dffb21564b285ff4f84; no voters:
I20260929 18:35:54.484244 3142 raft_consensus.cc:2781] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:54.484421 3142 raft_consensus.cc:686] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 1 LEADER]: Becoming Leader. State: Replica: adde224930134115a5f3e6137cbb57e0, State: Running, Role: LEADER
I20260929 18:35:54.484566 3142 consensus_queue.cc:237] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } }
I20260929 18:35:54.486497 2539 catalog_manager.cc:5495] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to adde224930134115a5f3e6137cbb57e0 (127.0.198.1). New cstate: current_term: 1 leader_uuid: "adde224930134115a5f3e6137cbb57e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adde224930134115a5f3e6137cbb57e0" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:54.513502 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:54.510101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.0.198.1:41867)
0929 18:35:54.510220 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:54.510225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:54.510281 (+ 56us) server_negotiation.cc:407] Connection header received
0929 18:35:54.510428 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:54.510431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:54.510485 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:54.510563 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:54.510729 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.511497 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.512393 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:54.512694 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:54.512809 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:54.513011 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:54.513096 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:54.513158 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:54.513301 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 18:35:54.513361 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:35:54.520462 2976 raft_consensus.cc:1261] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Refusing update from remote peer adde224930134115a5f3e6137cbb57e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:54.520632 2835 raft_consensus.cc:1261] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Refusing update from remote peer adde224930134115a5f3e6137cbb57e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:54.520869 3142 consensus_queue.cc:1035] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fa85fcc314dffb21564b285ff4f84" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:54.521039 3164 consensus_queue.cc:1035] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "136c1fbf4742488faad05863f6b0ccf5" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:35:54.537418 3109 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:35:56.089541 2539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:56.089706 2539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:56.092154 2539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 813c6d3abbc94d7e94e9cd6624ec663a: Sending DeleteTablet for 3 replicas of tablet 8659acf4cb49495fbfa38fd2f01ae42a
I20260929 18:35:56.092767 2676 tablet_service.cc:1514] Processing DeleteTablet for tablet 8659acf4cb49495fbfa38fd2f01ae42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:56 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33408
I20260929 18:35:56.092857 2816 tablet_service.cc:1514] Processing DeleteTablet for tablet 8659acf4cb49495fbfa38fd2f01ae42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:56 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260929 18:35:56.092968 2956 tablet_service.cc:1514] Processing DeleteTablet for tablet 8659acf4cb49495fbfa38fd2f01ae42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:56 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:48184
I20260929 18:35:56.093324 3191 tablet_replica.cc:330] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0: stopping tablet replica
I20260929 18:35:56.093317 3192 tablet_replica.cc:330] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: stopping tablet replica
I20260929 18:35:56.093448 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 2611
I20260929 18:35:56.093447 3192 raft_consensus.cc:2229] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:56.093449 3191 raft_consensus.cc:2229] T 8659acf4cb49495fbfa38fd2f01ae42a P adde224930134115a5f3e6137cbb57e0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:56.094101 3192 raft_consensus.cc:2258] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:56.094210 3193 tablet_replica.cc:330] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: stopping tablet replica
I20260929 18:35:56.094322 3193 raft_consensus.cc:2229] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:56.094424 3193 raft_consensus.cc:2258] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:56.094751 3192 ts_tablet_manager.cc:1905] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:56.095079 3193 ts_tablet_manager.cc:1905] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:56.097368 3193 ts_tablet_manager.cc:1918] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:35:56.097434 3193 log.cc:1196] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/8659acf4cb49495fbfa38fd2f01ae42a
I20260929 18:35:56.097636 3193 ts_tablet_manager.cc:1939] T 8659acf4cb49495fbfa38fd2f01ae42a P db3fa85fcc314dffb21564b285ff4f84: Deleting consensus metadata
I20260929 18:35:56.098519 2526 catalog_manager.cc:4841] TS db3fa85fcc314dffb21564b285ff4f84 (127.0.198.3:38037): tablet 8659acf4cb49495fbfa38fd2f01ae42a (table test-table [id=1f597b8c8cb842df9f945e0f8c564f69]) successfully deleted
I20260929 18:35:56.099426 3192 ts_tablet_manager.cc:1918] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:35:56.099510 3192 log.cc:1196] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/8659acf4cb49495fbfa38fd2f01ae42a
I20260929 18:35:56.099747 3192 ts_tablet_manager.cc:1939] T 8659acf4cb49495fbfa38fd2f01ae42a P 136c1fbf4742488faad05863f6b0ccf5: Deleting consensus metadata
I20260929 18:35:56.100476 2525 catalog_manager.cc:4841] TS 136c1fbf4742488faad05863f6b0ccf5 (127.0.198.2:42117): tablet 8659acf4cb49495fbfa38fd2f01ae42a (table test-table [id=1f597b8c8cb842df9f945e0f8c564f69]) successfully deleted
W20260929 18:35:56.101274 2524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:41867 (error 108)
I20260929 18:35:56.101475 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 2751
I20260929 18:35:56.103040 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:56.102648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41867 (local address 127.0.0.1:43232)
0929 18:35:56.102888 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:56.102954 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
W20260929 18:35:56.103176 2524 catalog_manager.cc:4584] TS adde224930134115a5f3e6137cbb57e0 (127.0.198.1:41867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8659acf4cb49495fbfa38fd2f01ae42a: Network error: Client connection negotiation failed: client connection to 127.0.198.1:41867: connect: Connection refused (error 111)
I20260929 18:35:56.107817 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 2891
I20260929 18:35:56.113631 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 2507
2026-09-29T18:35:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6185 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:56 dist-test-slave-q5qx krb5kdc[3198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:56 dist-test-slave-q5qx krb5kdc[3198](info): set up 2 sockets
Sep 29 18:35:56 dist-test-slave-q5qx krb5kdc[3198](info): commencing operation
krb5kdc: starting...
W20260929 18:35:57.077770 792 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:35:57 dist-test-slave-q5qx krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:35:57.197492 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:33189
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:33189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:57.309741 3211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:57.310113 3211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:57.310161 3211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:57.314242 3211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:57.314309 3211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:57.314327 3211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:57.314343 3211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:57.314356 3211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:57.319969 3211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:33189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:33189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:57.321072 3211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:57.322110 3211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:57.331374 3217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:57.331611 3211 server_base.cc:1034] running on GCE node
W20260929 18:35:57.331504 3216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:57.331363 3219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:57.332382 3211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:57.333575 3211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:57.334758 3211 hybrid_clock.cc:648] HybridClock initialized: now 1790706957334680 us; error 96 us; skew 500 ppm
Sep 29 18:35:57 dist-test-slave-q5qx krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:57.337669 3211 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:35:57.338676 3211 webserver.cc:460] Webserver started at http://127.0.198.62:45035/ using document root <none> and password file <none>
I20260929 18:35:57.339306 3211 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:57.339377 3211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:57.339576 3211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:57.341604 3211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "84cfe966e98b4b2fbb7d107f353c4ae4"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "b9ab8ad4ece31232a7a406c0c48ed49f"
server_key_iv: "a5fe69eb01169aceca4ef71d4bd4fcf2"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.342185 3211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "84cfe966e98b4b2fbb7d107f353c4ae4"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "b9ab8ad4ece31232a7a406c0c48ed49f"
server_key_iv: "a5fe69eb01169aceca4ef71d4bd4fcf2"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.346129 3211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:35:57.348717 3226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:57.350322 3211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:57.350462 3211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "84cfe966e98b4b2fbb7d107f353c4ae4"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "b9ab8ad4ece31232a7a406c0c48ed49f"
server_key_iv: "a5fe69eb01169aceca4ef71d4bd4fcf2"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.350601 3211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:57.360471 3211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:57.363662 3211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:57.363849 3211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:57.370769 3211 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:33189
I20260929 18:35:57.370823 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:33189 every 8 connection(s)
I20260929 18:35:57.371724 3211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:57.373826 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 3211
I20260929 18:35:57.373940 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:35:57.374228 792 external_mini_cluster.cc:1428] Setting key 9381a0fec6c938188d8e2ceaeea4feb5
I20260929 18:35:57.374708 3279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:57.380744 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4: Bootstrap starting.
Sep 29 18:35:57 dist-test-slave-q5qx krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:57.383235 3279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:57.383988 3279 log.cc:824] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:57.386004 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4: No bootstrap required, opened a new log
I20260929 18:35:57.388098 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:57.375608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.0.198.62:33189)
0929 18:35:57.376079 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:57.376094 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:57.376130 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:57.376772 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:57.376790 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:57.377134 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:57.377441 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:57.378240 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.379249 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.380116 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.380592 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.380743 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:57.383289 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:57.383296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:57.383314 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:57.383350 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:57.385088 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:57.385500 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:57.385505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:57.385510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:57.385620 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:57.385876 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:57.385879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:57.385880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:57.386371 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:57.386557 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:57.386834 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 18:35:57.387154 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":150,"threads_started":1}
I20260929 18:35:57.390214 3279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33189 } }
I20260929 18:35:57.390586 3279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:57.390652 3279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cfe966e98b4b2fbb7d107f353c4ae4, State: Initialized, Role: FOLLOWER
I20260929 18:35:57.391151 3279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33189 } }
I20260929 18:35:57.391315 3279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:57.391379 3279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:57.391486 3279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:57.392659 3279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33189 } }
I20260929 18:35:57.393028 3279 leader_election.cc:302] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84cfe966e98b4b2fbb7d107f353c4ae4; no voters:
I20260929 18:35:57.393369 3279 leader_election.cc:288] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:57.393548 3284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:57.393756 3284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [term 1 LEADER]: Becoming Leader. State: Replica: 84cfe966e98b4b2fbb7d107f353c4ae4, State: Running, Role: LEADER
I20260929 18:35:57.394099 3284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33189 } }
I20260929 18:35:57.394721 3279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:57.395385 3285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84cfe966e98b4b2fbb7d107f353c4ae4. Latest consensus state: current_term: 1 leader_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33189 } } }
I20260929 18:35:57.395514 3285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:57.395946 3286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84cfe966e98b4b2fbb7d107f353c4ae4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33189 } } }
I20260929 18:35:57.396049 3286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4 [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:57.399603 3299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:57.399688 3299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:57.399755 3290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:57.400449 3290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:57.406225 3290 catalog_manager.cc:1347] Generated new cluster ID: cb4779da4cea4f6cb215486c82581c08
I20260929 18:35:57.406322 3290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:57.419929 3290 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:57.420904 3290 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:57.426920 3290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84cfe966e98b4b2fbb7d107f353c4ae4: Generated new TSK 0
I20260929 18:35:57.427632 3290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:35:57.461719 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:33189
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:57.577294 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:57.577577 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:57.577620 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:57.581621 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:57.581684 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:57.581741 3305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:35:57.587028 3305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:33189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:57.588369 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:57.589471 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:57.599581 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:57.599611 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:57.599588 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:57.600108 3305 server_base.cc:1034] running on GCE node
I20260929 18:35:57.600746 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:57.601656 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:57.602816 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790706957602767 us; error 64 us; skew 500 ppm
Sep 29 18:35:57 dist-test-slave-q5qx krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:57.605603 3305 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:35:57.606606 3305 webserver.cc:460] Webserver started at http://127.0.198.1:46219/ using document root <none> and password file <none>
I20260929 18:35:57.607162 3305 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:57.607221 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:57.607424 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:57.609474 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "b55b720d7a944682a1cc60370cb589e7"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "ebd6126879e3e2c5e76812a30f36b662"
server_key_iv: "b2e6d7fa1cabf5bac9fea3b61878ff4d"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.610052 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "b55b720d7a944682a1cc60370cb589e7"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "ebd6126879e3e2c5e76812a30f36b662"
server_key_iv: "b2e6d7fa1cabf5bac9fea3b61878ff4d"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.613747 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:57.616277 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:57.617508 3305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:57.617623 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "b55b720d7a944682a1cc60370cb589e7"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "ebd6126879e3e2c5e76812a30f36b662"
server_key_iv: "b2e6d7fa1cabf5bac9fea3b61878ff4d"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.617733 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:57.634060 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:57.637310 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:57.637490 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:57.638154 3305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:57.639014 3305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:57.639062 3305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:57.639114 3305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:57.639144 3305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:57.647167 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:39559
I20260929 18:35:57.647207 3433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:39559 every 8 connection(s)
I20260929 18:35:57.648070 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:57.648526 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 3305
I20260929 18:35:57.648633 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:57.648907 792 external_mini_cluster.cc:1428] Setting key c1fc384253c9c8efcd423889251c9c48
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
Sep 29 18:35:57 dist-test-slave-q5qx krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:57.665258 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:57.650113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:44919 (local address 127.0.198.62:33189)
0929 18:35:57.650239 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:57.650244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:57.651272 (+ 1028us) server_negotiation.cc:407] Connection header received
0929 18:35:57.652625 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:57.652629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:57.652682 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:57.652757 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:57.654063 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.654647 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.655801 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.656037 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.656124 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:57.661250 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:57.661254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:57.661256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:57.661289 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:57.662783 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:57.663434 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:57.663437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:57.663438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:57.663527 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:57.663966 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:57.663969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:57.663970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:57.664209 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:57.664299 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:57.665010 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 18:35:57.665134 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:35:57.666436 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:57.650627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:33189 (local address 127.0.198.1:44919)
0929 18:35:57.651130 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:57.651173 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:57.652429 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:57.652972 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:57.652979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:57.653329 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:57.653861 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:57.653876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.654760 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.654763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:57.655684 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:57.655693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.656172 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.656176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:57.656344 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:57.656885 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:57.656901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:57.661039 (+ 4138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:57.662913 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:57.662920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:57.662939 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:57.663295 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:57.663615 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:57.663617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:57.663619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:57.663810 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:57.664324 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:57.664331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:57.664689 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 18:35:57.664920 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 18:35:57.665294 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":150,"threads_started":1}
I20260929 18:35:57.667781 3434 heartbeater.cc:347] Connected to a master server at 127.0.198.62:33189
I20260929 18:35:57.668041 3434 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:57.668555 3434 heartbeater.cc:511] Master 127.0.198.62:33189 requested a full tablet report, sending...
I20260929 18:35:57.670346 3243 ts_manager.cc:194] Registered new tserver with Master: b55b720d7a944682a1cc60370cb589e7 (127.0.198.1:39559)
I20260929 18:35:57.671592 3243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:44919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:35:57.677629 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:33189
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:57.786774 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:57.787070 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:57.787129 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:57.791318 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:57.791391 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:57.791479 3439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:35:57.796995 3439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:33189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:57.798098 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:57.799101 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:57.808632 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:57.808712 3439 server_base.cc:1034] running on GCE node
W20260929 18:35:57.808647 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:57.808651 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:57.809394 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:57.810081 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:57.811270 3439 hybrid_clock.cc:648] HybridClock initialized: now 1790706957811255 us; error 29 us; skew 500 ppm
Sep 29 18:35:57 dist-test-slave-q5qx krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:57.814314 3439 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:35:57.815313 3439 webserver.cc:460] Webserver started at http://127.0.198.2:35073/ using document root <none> and password file <none>
I20260929 18:35:57.815994 3439 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:57.816072 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:57.816308 3439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:57.818409 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "295bb66476b244759bb753be40cd0a3b"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "57b89d58c507bff4026e47468663a8e0"
server_key_iv: "292c9f78986b1786707542d924d5398d"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.818913 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "295bb66476b244759bb753be40cd0a3b"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "57b89d58c507bff4026e47468663a8e0"
server_key_iv: "292c9f78986b1786707542d924d5398d"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.822721 3439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:35:57.825131 3454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:57.826453 3439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:57.826551 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "295bb66476b244759bb753be40cd0a3b"
format_stamp: "Formatted at 2026-09-29 18:35:57 on dist-test-slave-q5qx"
server_key: "57b89d58c507bff4026e47468663a8e0"
server_key_iv: "292c9f78986b1786707542d924d5398d"
server_key_version: "encryptionkey@0"
I20260929 18:35:57.826643 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:57.847697 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:57.850898 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:57.851065 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:57.851625 3439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:57.852452 3439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:57.852499 3439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:57.852538 3439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:57.852558 3439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:57.861176 3439 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42479
I20260929 18:35:57.861196 3567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42479 every 8 connection(s)
I20260929 18:35:57.862206 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:57.864104 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 3439
I20260929 18:35:57.864216 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:57.864501 792 external_mini_cluster.cc:1428] Setting key 7d92b772ef2d95de28446d6cac4982ca
Sep 29 18:35:57 dist-test-slave-q5qx krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:35:57.876389 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:57.864190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:47445 (local address 127.0.198.62:33189)
0929 18:35:57.864333 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:57.864338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:57.865230 (+ 892us) server_negotiation.cc:407] Connection header received
0929 18:35:57.866747 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:57.866758 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:57.866813 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:57.866899 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:57.868097 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.868615 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.869749 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.869962 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.870084 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:57.872808 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:57.872811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:57.872814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:57.872839 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:57.874319 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:57.874853 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:57.874857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:57.874859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:57.874955 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:57.875318 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:57.875321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:57.875322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:57.875500 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:57.875584 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:57.876093 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 18:35:57.876215 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:35:57.877209 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:57.864601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:33189 (local address 127.0.198.2:47445)
0929 18:35:57.865084 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:57.865137 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:57.866531 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:57.867066 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:57.867074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:57.867429 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:57.867952 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:57.867967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.868751 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.868755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:57.869630 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:57.869639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:57.870122 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:57.870126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:57.870294 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:57.871002 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:57.871019 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:57.872632 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:57.874455 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:57.874462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:57.874480 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:57.874741 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:57.875047 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:57.875049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:57.875051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:57.875208 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:57.875598 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:57.875606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:57.875865 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 18:35:57.876035 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 18:35:57.876373 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":133,"threads_started":1}
I20260929 18:35:57.878336 3568 heartbeater.cc:347] Connected to a master server at 127.0.198.62:33189
I20260929 18:35:57.878610 3568 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:57.879235 3568 heartbeater.cc:511] Master 127.0.198.62:33189 requested a full tablet report, sending...
I20260929 18:35:57.880491 3243 ts_manager.cc:194] Registered new tserver with Master: 295bb66476b244759bb753be40cd0a3b (127.0.198.2:42479)
I20260929 18:35:57.881057 3243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:47445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:35:57.890609 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:33189
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:58.001891 3573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:58.002331 3573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:58.002424 3573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:58.006935 3573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:58.007010 3573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:58.007090 3573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:35:58.012538 3573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:33189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:58.013584 3573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:58.014631 3573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:35:58.024703 3573 server_base.cc:1034] running on GCE node
W20260929 18:35:58.024619 3581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:58.024585 3578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:58.024613 3579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:58.025302 3573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:58.025991 3573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:58.027190 3573 hybrid_clock.cc:648] HybridClock initialized: now 1790706958027158 us; error 47 us; skew 500 ppm
Sep 29 18:35:58 dist-test-slave-q5qx krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:58.030177 3573 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:35:58.031123 3573 webserver.cc:460] Webserver started at http://127.0.198.3:34195/ using document root <none> and password file <none>
I20260929 18:35:58.031697 3573 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:58.031761 3573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:58.031970 3573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:58.034086 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "c26ed04c38734d788928416a10a3a05a"
format_stamp: "Formatted at 2026-09-29 18:35:58 on dist-test-slave-q5qx"
server_key: "5e2a22d2ab003b503c4d17ea33194474"
server_key_iv: "6b09260a31aabc327aae39ca9231845e"
server_key_version: "encryptionkey@0"
I20260929 18:35:58.034642 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "c26ed04c38734d788928416a10a3a05a"
format_stamp: "Formatted at 2026-09-29 18:35:58 on dist-test-slave-q5qx"
server_key: "5e2a22d2ab003b503c4d17ea33194474"
server_key_iv: "6b09260a31aabc327aae39ca9231845e"
server_key_version: "encryptionkey@0"
I20260929 18:35:58.038439 3573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:35:58.041061 3588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:58.042433 3573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:58.042546 3573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "c26ed04c38734d788928416a10a3a05a"
format_stamp: "Formatted at 2026-09-29 18:35:58 on dist-test-slave-q5qx"
server_key: "5e2a22d2ab003b503c4d17ea33194474"
server_key_iv: "6b09260a31aabc327aae39ca9231845e"
server_key_version: "encryptionkey@0"
I20260929 18:35:58.042654 3573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:58.053553 3573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:58.056805 3573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:58.056988 3573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:58.057576 3573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:58.058425 3573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:58.058472 3573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:58.058525 3573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:58.058553 3573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:58.067243 3573 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:42987
I20260929 18:35:58.067271 3701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:42987 every 8 connection(s)
I20260929 18:35:58.068158 3573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:58.076898 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 3573
I20260929 18:35:58.077015 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:58.077317 792 external_mini_cluster.cc:1428] Setting key 740008f8812a117a16673dc019336e5e
Sep 29 18:35:58 dist-test-slave-q5qx krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:58.082897 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:58.070343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:49401 (local address 127.0.198.62:33189)
0929 18:35:58.070486 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:58.070490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:58.071393 (+ 903us) server_negotiation.cc:407] Connection header received
0929 18:35:58.072852 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:58.072856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:58.072913 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:58.072990 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:58.074256 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:58.074780 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:58.076291 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:58.076515 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:58.076603 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:58.079395 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:58.079399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:58.079402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:58.079430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:58.080815 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:58.081350 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:58.081354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:58.081356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:58.081435 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:58.081752 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:58.081755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:58.081757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:58.081957 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:58.082068 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:58.082656 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 18:35:58.082764 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:35:58.083874 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:58.070771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:33189 (local address 127.0.198.3:49401)
0929 18:35:58.071234 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:58.071278 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:58.072621 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:58.073191 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:58.073198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:58.073562 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:58.074114 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:58.074131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:58.074905 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:58.074908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:58.076180 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:58.076190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:58.076660 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:58.076664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:58.076856 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:58.077514 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:58.077531 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:58.079220 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:58.080926 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:58.080932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:58.080951 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:58.081245 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:58.081514 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:58.081517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:58.081519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:58.081666 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:58.082105 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:58.082111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:58.082398 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 18:35:58.082595 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 18:35:58.082984 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":129,"threads_started":1}
I20260929 18:35:58.085035 3702 heartbeater.cc:347] Connected to a master server at 127.0.198.62:33189
I20260929 18:35:58.085246 3702 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:58.085736 3702 heartbeater.cc:511] Master 127.0.198.62:33189 requested a full tablet report, sending...
I20260929 18:35:58.086802 3243 ts_manager.cc:194] Registered new tserver with Master: c26ed04c38734d788928416a10a3a05a (127.0.198.3:42987)
I20260929 18:35:58.087378 3243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:49401
I20260929 18:35:58.091176 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:35:58.103395 3282 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:58.101830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.0.198.62:33189)
0929 18:35:58.102035 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:58.102040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:58.102189 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:35:58.102322 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:58.102325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:58.102377 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:58.102448 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:58.103248 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40830: BlockingRecv error: recv got EOF from 127.0.0.1:40830 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 18:35:58.104673 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 3305
I20260929 18:35:58.109447 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 3439
I20260929 18:35:58.114205 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 3573
I20260929 18:35:58.118913 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 3211
2026-09-29T18:35:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2003 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:58 dist-test-slave-q5qx krb5kdc[3714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:58 dist-test-slave-q5qx krb5kdc[3714](info): set up 2 sockets
Sep 29 18:35:58 dist-test-slave-q5qx krb5kdc[3714](info): commencing operation
krb5kdc: starting...
W20260929 18:35:59.077734 792 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:35:59 dist-test-slave-q5qx krb5kdc[3714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:35:59.197540 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44667
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:59.316784 3727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:59.317068 3727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:59.317124 3727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:59.321200 3727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:59.321269 3727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:59.321296 3727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:59.321316 3727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:59.321337 3727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:59.326740 3727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:59.327823 3727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:59.328859 3727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:59.337308 3732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:59.337338 3733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:59.337311 3735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:59.337433 3727 server_base.cc:1034] running on GCE node
I20260929 18:35:59.338207 3727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:59.339362 3727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:59.340529 3727 hybrid_clock.cc:648] HybridClock initialized: now 1790706959340499 us; error 45 us; skew 500 ppm
Sep 29 18:35:59 dist-test-slave-q5qx krb5kdc[3714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:59.343457 3727 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:35:59.344328 3727 webserver.cc:460] Webserver started at http://127.0.198.62:33535/ using document root <none> and password file <none>
I20260929 18:35:59.344846 3727 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:59.344899 3727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:59.345067 3727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:59.347111 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "2f15a3bcd67143cc8607f0bebf809da9"
format_stamp: "Formatted at 2026-09-29 18:35:59 on dist-test-slave-q5qx"
server_key: "376de55bc9c5ffa3d73ad59b1b44e2d7"
server_key_iv: "3e3da8f6cbccd10d578586126ffa6d6b"
server_key_version: "encryptionkey@0"
I20260929 18:35:59.347616 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "2f15a3bcd67143cc8607f0bebf809da9"
format_stamp: "Formatted at 2026-09-29 18:35:59 on dist-test-slave-q5qx"
server_key: "376de55bc9c5ffa3d73ad59b1b44e2d7"
server_key_iv: "3e3da8f6cbccd10d578586126ffa6d6b"
server_key_version: "encryptionkey@0"
I20260929 18:35:59.351377 3727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:35:59.353921 3742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:59.355255 3727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:59.355360 3727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "2f15a3bcd67143cc8607f0bebf809da9"
format_stamp: "Formatted at 2026-09-29 18:35:59 on dist-test-slave-q5qx"
server_key: "376de55bc9c5ffa3d73ad59b1b44e2d7"
server_key_iv: "3e3da8f6cbccd10d578586126ffa6d6b"
server_key_version: "encryptionkey@0"
I20260929 18:35:59.355453 3727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:59.378386 3727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:59.381524 3727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:59.381683 3727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:59.388867 3727 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:44667
I20260929 18:35:59.389199 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:44667 every 8 connection(s)
I20260929 18:35:59.390044 3727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:59.393213 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 3727
I20260929 18:35:59.393378 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:35:59.393749 792 external_mini_cluster.cc:1428] Setting key 1d47cf71e3efd589fd10ffb1316ec8fd
I20260929 18:35:59.394057 3795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:59.402402 3795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9: Bootstrap starting.
Sep 29 18:35:59 dist-test-slave-q5qx krb5kdc[3714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:35:59.404997 3795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:59.405705 3795 log.cc:824] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:59.407950 3795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9: No bootstrap required, opened a new log
I20260929 18:35:59.409765 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:59.395192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.0.198.62:44667)
0929 18:35:59.395752 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:59.395773 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:59.395836 (+ 63us) server_negotiation.cc:407] Connection header received
0929 18:35:59.396945 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:59.396992 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:59.397449 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:59.397839 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:59.399202 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:59.400747 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:59.401736 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:59.402365 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:59.402571 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:59.404948 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:59.404953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:59.404966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:59.404992 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:59.406814 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:59.407324 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:59.407328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:59.407333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:59.407434 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:59.407706 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:59.407708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:59.407709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:59.408110 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:59.408264 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:59.408561 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 18:35:59.408843 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":142,"threads_started":1}
I20260929 18:35:59.411329 3795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f15a3bcd67143cc8607f0bebf809da9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44667 } }
I20260929 18:35:59.411598 3795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:59.411653 3795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f15a3bcd67143cc8607f0bebf809da9, State: Initialized, Role: FOLLOWER
I20260929 18:35:59.412089 3795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f15a3bcd67143cc8607f0bebf809da9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44667 } }
I20260929 18:35:59.412237 3795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:59.412302 3795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:59.412392 3795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:59.412675 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:59.413331 3795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f15a3bcd67143cc8607f0bebf809da9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44667 } }
I20260929 18:35:59.413652 3795 leader_election.cc:302] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f15a3bcd67143cc8607f0bebf809da9; no voters:
I20260929 18:35:59.413941 3795 leader_election.cc:288] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:59.414067 3800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:59.414235 3800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f15a3bcd67143cc8607f0bebf809da9, State: Running, Role: LEADER
I20260929 18:35:59.414539 3800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f15a3bcd67143cc8607f0bebf809da9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44667 } }
I20260929 18:35:59.415272 3795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:59.415737 3803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f15a3bcd67143cc8607f0bebf809da9. Latest consensus state: current_term: 1 leader_uuid: "2f15a3bcd67143cc8607f0bebf809da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f15a3bcd67143cc8607f0bebf809da9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44667 } } }
I20260929 18:35:59.415931 3803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:59.416173 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f15a3bcd67143cc8607f0bebf809da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f15a3bcd67143cc8607f0bebf809da9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44667 } } }
I20260929 18:35:59.416280 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:59.416437 3807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:59.419461 3807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:59.424808 3807 catalog_manager.cc:1347] Generated new cluster ID: 84993928bbc5407ab43638454a4a8a00
I20260929 18:35:59.424880 3807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:59.432998 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:59.427445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.198.62:44667)
0929 18:35:59.427550 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:59.427554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:59.427851 (+ 297us) server_negotiation.cc:407] Connection header received
0929 18:35:59.427983 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:59.427986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:59.428028 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:59.428112 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:59.428716 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:59.429489 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:59.430724 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:59.431078 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:59.431165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:59.432037 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:59.432041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:59.432044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:59.432580 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:59.432612 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:59.432629 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:59.432705 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:59.432799 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:35:59.432856 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:35:59.437515 3807 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:59.438537 3807 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:59.443591 3807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f15a3bcd67143cc8607f0bebf809da9: Generated new TSK 0
I20260929 18:35:59.444284 3807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:59.452687 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 3727
2026-09-29T18:35:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1332 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:59 dist-test-slave-q5qx krb5kdc[3828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:59 dist-test-slave-q5qx krb5kdc[3828](info): set up 2 sockets
Sep 29 18:35:59 dist-test-slave-q5qx krb5kdc[3828](info): commencing operation
krb5kdc: starting...
W20260929 18:36:00.405733 792 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:00 dist-test-slave-q5qx krb5kdc[3828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:36:00.525202 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44675
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:00.635708 3841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:00.635993 3841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:00.636047 3841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:00.640193 3841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:00.640259 3841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:00.640285 3841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:00.640307 3841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:00.640333 3841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:00.645846 3841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:44675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:00.646935 3841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:00.647949 3841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:00.657764 3846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:00.657853 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:00.657758 3847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:00.658635 3841 server_base.cc:1034] running on GCE node
I20260929 18:36:00.659356 3841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:00.660794 3841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:00.662015 3841 hybrid_clock.cc:648] HybridClock initialized: now 1790706960661982 us; error 40 us; skew 500 ppm
Sep 29 18:36:00 dist-test-slave-q5qx krb5kdc[3828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:00.665375 3841 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:36:00.666579 3841 webserver.cc:460] Webserver started at http://127.0.198.62:46509/ using document root <none> and password file <none>
I20260929 18:36:00.667302 3841 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:00.667387 3841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:00.667608 3841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:00.670622 3841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "0418a4b393f24832a983929d40d565d5"
format_stamp: "Formatted at 2026-09-29 18:36:00 on dist-test-slave-q5qx"
server_key: "a9e3a2f79c6a4c3b1c4d4b9421c75e0e"
server_key_iv: "0ef1d4bb07ab7a3cc4e433041e5bd802"
server_key_version: "encryptionkey@0"
I20260929 18:36:00.671316 3841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "0418a4b393f24832a983929d40d565d5"
format_stamp: "Formatted at 2026-09-29 18:36:00 on dist-test-slave-q5qx"
server_key: "a9e3a2f79c6a4c3b1c4d4b9421c75e0e"
server_key_iv: "0ef1d4bb07ab7a3cc4e433041e5bd802"
server_key_version: "encryptionkey@0"
I20260929 18:36:00.676328 3841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 18:36:00.679585 3856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:00.681164 3841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:36:00.681311 3841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "0418a4b393f24832a983929d40d565d5"
format_stamp: "Formatted at 2026-09-29 18:36:00 on dist-test-slave-q5qx"
server_key: "a9e3a2f79c6a4c3b1c4d4b9421c75e0e"
server_key_iv: "0ef1d4bb07ab7a3cc4e433041e5bd802"
server_key_version: "encryptionkey@0"
I20260929 18:36:00.681450 3841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:00.728715 3841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:00.731712 3841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:00.731868 3841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:00.738632 3841 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:44675
I20260929 18:36:00.738668 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:44675 every 8 connection(s)
I20260929 18:36:00.739615 3841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:00.741885 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 3841
I20260929 18:36:00.742022 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:00.742292 792 external_mini_cluster.cc:1428] Setting key 83c988ddb6406611366761be0bed7424
I20260929 18:36:00.742496 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:00.749231 3909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5: Bootstrap starting.
Sep 29 18:36:00 dist-test-slave-q5qx krb5kdc[3828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:00.751657 3909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:00.752369 3909 log.cc:824] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:00.754302 3909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5: No bootstrap required, opened a new log
I20260929 18:36:00.756716 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:00.743633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.198.62:44675)
0929 18:36:00.744026 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:00.744041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:00.744075 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:36:00.745018 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:00.745037 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:00.745374 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:00.745660 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:00.746569 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:00.747582 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:00.748805 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:00.749290 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:00.749444 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:00.751977 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:00.751982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:00.751997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:00.752031 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:00.753656 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:00.754085 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:00.754090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:00.754094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:00.754203 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:00.754483 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:00.754487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:00.754489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:00.755000 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:00.755185 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:00.755441 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 18:36:00.755729 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260929 18:36:00.758358 3909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418a4b393f24832a983929d40d565d5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44675 } }
I20260929 18:36:00.758574 3909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:00.758610 3909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0418a4b393f24832a983929d40d565d5, State: Initialized, Role: FOLLOWER
I20260929 18:36:00.759037 3909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418a4b393f24832a983929d40d565d5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44675 } }
I20260929 18:36:00.759171 3909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:00.759227 3909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:00.759320 3909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:00.760181 3909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418a4b393f24832a983929d40d565d5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44675 } }
I20260929 18:36:00.760496 3909 leader_election.cc:302] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0418a4b393f24832a983929d40d565d5; no voters:
I20260929 18:36:00.760762 3909 leader_election.cc:288] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:00.760882 3914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:00.761090 3914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0418a4b393f24832a983929d40d565d5, State: Running, Role: LEADER
I20260929 18:36:00.761693 3914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418a4b393f24832a983929d40d565d5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44675 } }
I20260929 18:36:00.761950 3909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:00.763409 3916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0418a4b393f24832a983929d40d565d5. Latest consensus state: current_term: 1 leader_uuid: "0418a4b393f24832a983929d40d565d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418a4b393f24832a983929d40d565d5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44675 } } }
I20260929 18:36:00.763530 3916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:00.763738 3915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0418a4b393f24832a983929d40d565d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418a4b393f24832a983929d40d565d5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44675 } } }
I20260929 18:36:00.763827 3915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:00.764283 3923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:00.766911 3923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:00.772259 3923 catalog_manager.cc:1347] Generated new cluster ID: f62d1a5441d449d9948c93de9bd89b51
I20260929 18:36:00.772423 3923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:00.787834 3923 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:00.788796 3923 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:00.796073 3923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0418a4b393f24832a983929d40d565d5: Generated new TSK 0
I20260929 18:36:00.796790 3923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:36:00.829087 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44675
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:00.940745 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:00.941017 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:00.941079 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:00.945077 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:00.945135 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:00.945214 3935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:00.950464 3935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:00.951479 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:00.952442 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:00.961812 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:00.961822 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:00.961905 3935 server_base.cc:1034] running on GCE node
W20260929 18:36:00.961834 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:00.962564 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:00.963230 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:00.964411 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790706960964376 us; error 46 us; skew 500 ppm
Sep 29 18:36:00 dist-test-slave-q5qx krb5kdc[3828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:00.967166 3935 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:36:00.968005 3935 webserver.cc:460] Webserver started at http://127.0.198.1:39277/ using document root <none> and password file <none>
I20260929 18:36:00.968542 3935 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:00.968607 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:00.968789 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:00.970834 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "e889d0d0863a4a74abf1d41386eaa1cb"
format_stamp: "Formatted at 2026-09-29 18:36:00 on dist-test-slave-q5qx"
server_key: "9f9ab0b89e375a673e7b39551e59dffb"
server_key_iv: "65a99968cad53e9b8291d07e3de67c52"
server_key_version: "encryptionkey@0"
I20260929 18:36:00.971364 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "e889d0d0863a4a74abf1d41386eaa1cb"
format_stamp: "Formatted at 2026-09-29 18:36:00 on dist-test-slave-q5qx"
server_key: "9f9ab0b89e375a673e7b39551e59dffb"
server_key_iv: "65a99968cad53e9b8291d07e3de67c52"
server_key_version: "encryptionkey@0"
I20260929 18:36:00.975003 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:00.977388 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:00.978749 3935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:00.978886 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "e889d0d0863a4a74abf1d41386eaa1cb"
format_stamp: "Formatted at 2026-09-29 18:36:00 on dist-test-slave-q5qx"
server_key: "9f9ab0b89e375a673e7b39551e59dffb"
server_key_iv: "65a99968cad53e9b8291d07e3de67c52"
server_key_version: "encryptionkey@0"
I20260929 18:36:00.978988 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:01.006016 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:01.009250 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:01.009438 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:01.010061 3935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:01.010898 3935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:01.010946 3935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.010999 3935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:01.011029 3935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.019579 3935 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:45081
I20260929 18:36:01.019608 4063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:45081 every 8 connection(s)
I20260929 18:36:01.020462 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:01.025739 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 3935
I20260929 18:36:01.025852 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:01.026158 792 external_mini_cluster.cc:1428] Setting key b5b09a92b41d704d1451137f3473f5d1
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[3828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:36:01.035177 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:01.022347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:54801 (local address 127.0.198.62:44675)
0929 18:36:01.022529 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:01.022533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:01.023356 (+ 823us) server_negotiation.cc:407] Connection header received
0929 18:36:01.024765 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:01.024769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:01.024823 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:01.024899 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:01.026127 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.026799 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.028314 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.028566 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.028648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:01.031247 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:01.031251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:01.031254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:01.031282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:01.032868 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.033351 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.033355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.033356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.033452 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.033744 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.033747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.033749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.034034 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:01.034163 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:01.034800 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 18:36:01.034929 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:36:01.036056 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:01.022741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44675 (local address 127.0.198.1:54801)
0929 18:36:01.023185 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:01.023230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:01.024560 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:01.025078 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:01.025085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:01.025448 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:01.025952 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:01.025995 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.026911 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.026914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:01.028189 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:01.028202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.028705 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.028709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:01.028910 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:01.029572 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:01.029588 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:01.031062 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:01.032984 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:01.032989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:01.033004 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:01.033257 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:01.033531 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:01.033533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:01.033534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:01.033662 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:01.034189 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:01.034195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:01.034550 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 18:36:01.034727 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:36:01.035016 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
I20260929 18:36:01.037367 4064 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:36:01.037662 4064 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:01.038275 4064 heartbeater.cc:511] Master 127.0.198.62:44675 requested a full tablet report, sending...
I20260929 18:36:01.039940 3873 ts_manager.cc:194] Registered new tserver with Master: e889d0d0863a4a74abf1d41386eaa1cb (127.0.198.1:45081)
I20260929 18:36:01.041072 3873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:54801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:36:01.053371 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44675
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:01.162611 4069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:01.162909 4069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:01.162969 4069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:01.167110 4069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:01.167173 4069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:01.167234 4069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:01.172672 4069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:01.173738 4069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:01.174827 4069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:01.184753 4075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:01.184863 4074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:01.184919 4069 server_base.cc:1034] running on GCE node
W20260929 18:36:01.184863 4077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:01.185575 4069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:01.186305 4069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:01.187464 4069 hybrid_clock.cc:648] HybridClock initialized: now 1790706961187441 us; error 35 us; skew 500 ppm
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[3828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:01.196817 4069 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:36:01.197796 4069 webserver.cc:460] Webserver started at http://127.0.198.2:33263/ using document root <none> and password file <none>
I20260929 18:36:01.198417 4069 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:01.198490 4069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:01.198680 4069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:01.200852 4069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "24241860886d499e85c2cb484e78b8ee"
format_stamp: "Formatted at 2026-09-29 18:36:01 on dist-test-slave-q5qx"
server_key: "a6ed2d7fe215ff57d4c802212e3bf5a7"
server_key_iv: "336ccebbf116163fe659774002f62084"
server_key_version: "encryptionkey@0"
I20260929 18:36:01.201422 4069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "24241860886d499e85c2cb484e78b8ee"
format_stamp: "Formatted at 2026-09-29 18:36:01 on dist-test-slave-q5qx"
server_key: "a6ed2d7fe215ff57d4c802212e3bf5a7"
server_key_iv: "336ccebbf116163fe659774002f62084"
server_key_version: "encryptionkey@0"
I20260929 18:36:01.205365 4069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:01.207966 4084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.209360 4069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:01.209491 4069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "24241860886d499e85c2cb484e78b8ee"
format_stamp: "Formatted at 2026-09-29 18:36:01 on dist-test-slave-q5qx"
server_key: "a6ed2d7fe215ff57d4c802212e3bf5a7"
server_key_iv: "336ccebbf116163fe659774002f62084"
server_key_version: "encryptionkey@0"
I20260929 18:36:01.209594 4069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:01.231946 4069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:01.235280 4069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:01.235468 4069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:01.236076 4069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:01.236896 4069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:01.236941 4069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.236990 4069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:01.237018 4069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.245672 4069 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42381
I20260929 18:36:01.245697 4197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42381 every 8 connection(s)
I20260929 18:36:01.246613 4069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:01.249855 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4069
I20260929 18:36:01.250005 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:01.250310 792 external_mini_cluster.cc:1428] Setting key 8cc70755c83fd57dfee2280b0411df8d
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[3828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:36:01.261067 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:01.248441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:60761 (local address 127.0.198.62:44675)
0929 18:36:01.248581 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:01.248586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:01.249522 (+ 936us) server_negotiation.cc:407] Connection header received
0929 18:36:01.251156 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:01.251159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:01.251203 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:01.251287 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:01.252665 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.253256 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.254511 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.254739 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.254837 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:01.257661 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:01.257665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:01.257667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:01.257690 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:01.259125 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.259588 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.259591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.259592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.259665 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.259994 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.259996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.259998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.260181 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:01.260255 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:01.260786 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 18:36:01.260927 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:36:01.261929 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:01.248926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44675 (local address 127.0.198.2:60761)
0929 18:36:01.249364 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:01.249419 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:01.250972 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:01.251439 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:01.251446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:01.251847 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:01.252518 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:01.252537 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.253434 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.253437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:01.254395 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:01.254404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.254900 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.254904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:01.255107 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:01.255769 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:01.255790 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:01.257470 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:01.259231 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:01.259236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:01.259249 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:01.259493 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:01.259769 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:01.259772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:01.259774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:01.259911 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:01.260285 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:01.260291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:01.260536 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 18:36:01.260693 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 18:36:01.260965 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260929 18:36:01.263157 4198 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44675
I20260929 18:36:01.263419 4198 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:01.264003 4198 heartbeater.cc:511] Master 127.0.198.62:44675 requested a full tablet report, sending...
I20260929 18:36:01.265086 3873 ts_manager.cc:194] Registered new tserver with Master: 24241860886d499e85c2cb484e78b8ee (127.0.198.2:42381)
I20260929 18:36:01.265704 3873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:60761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:36:01.277763 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44675
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:01.389096 4203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:01.389500 4203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:01.389566 4203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:01.395268 4203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:01.395361 4203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:01.395460 4203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:01.401434 4203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:01.402612 4203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:01.403618 4203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:01.413741 4208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:01.413743 4209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:01.413745 4211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:01.414327 4203 server_base.cc:1034] running on GCE node
I20260929 18:36:01.414813 4203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:01.415652 4203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:01.419054 4203 hybrid_clock.cc:648] HybridClock initialized: now 1790706961418993 us; error 84 us; skew 500 ppm
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[3828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:01.422816 4203 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:36:01.424064 4203 webserver.cc:460] Webserver started at http://127.0.198.3:39971/ using document root <none> and password file <none>
I20260929 18:36:01.424675 4203 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:01.424746 4203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:01.424966 4203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:01.427194 4203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "7de5d8e46d384f5481bcc9dcc10722af"
format_stamp: "Formatted at 2026-09-29 18:36:01 on dist-test-slave-q5qx"
server_key: "2bc227c36f75bb84f73cc5324d69c11a"
server_key_iv: "06038d4552a47678b4b80a9286399bd4"
server_key_version: "encryptionkey@0"
I20260929 18:36:01.427786 4203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "7de5d8e46d384f5481bcc9dcc10722af"
format_stamp: "Formatted at 2026-09-29 18:36:01 on dist-test-slave-q5qx"
server_key: "2bc227c36f75bb84f73cc5324d69c11a"
server_key_iv: "06038d4552a47678b4b80a9286399bd4"
server_key_version: "encryptionkey@0"
I20260929 18:36:01.431656 4203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:36:01.434269 4218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.435716 4203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:36:01.435851 4203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "7de5d8e46d384f5481bcc9dcc10722af"
format_stamp: "Formatted at 2026-09-29 18:36:01 on dist-test-slave-q5qx"
server_key: "2bc227c36f75bb84f73cc5324d69c11a"
server_key_iv: "06038d4552a47678b4b80a9286399bd4"
server_key_version: "encryptionkey@0"
I20260929 18:36:01.435973 4203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:01.450439 4203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:01.453589 4203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:01.453771 4203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:01.454389 4203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:01.455263 4203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:01.455310 4203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.455363 4203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:01.455390 4203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:01.463923 4203 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:33917
I20260929 18:36:01.463948 4331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:33917 every 8 connection(s)
I20260929 18:36:01.464836 4203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:01.474437 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4203
I20260929 18:36:01.474588 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:01.474887 792 external_mini_cluster.cc:1428] Setting key 01e80de9455f91aedd16ef186743eb30
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[3828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:01.479478 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:01.466746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:55607 (local address 127.0.198.62:44675)
0929 18:36:01.466921 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:01.466924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:01.467846 (+ 922us) server_negotiation.cc:407] Connection header received
0929 18:36:01.469265 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:01.469269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:01.469326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:01.469407 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:01.470923 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.471468 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.472663 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.472897 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.472991 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:01.475836 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:01.475840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:01.475843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:01.475874 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:01.477497 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.478007 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.478010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.478012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.478093 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.478417 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.478420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.478422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.478606 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:01.478688 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:01.479194 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 18:36:01.479318 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:36:01.480326 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:01.467123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44675 (local address 127.0.198.3:55607)
0929 18:36:01.467681 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:01.467731 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:01.469038 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:01.469649 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:01.469657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:01.470121 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:01.470742 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:01.470758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.471598 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.471601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:01.472530 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:01.472542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.473048 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.473052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:01.473218 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:01.473813 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:01.473835 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:01.475589 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:01.477602 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:01.477608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:01.477621 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:01.477907 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:01.478184 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:01.478186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:01.478188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:01.478327 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:01.478705 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:01.478712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:01.478967 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 18:36:01.479140 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 18:36:01.479500 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":125,"threads_started":1}
I20260929 18:36:01.481438 4332 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44675
I20260929 18:36:01.481686 4332 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:01.482234 4332 heartbeater.cc:511] Master 127.0.198.62:44675 requested a full tablet report, sending...
I20260929 18:36:01.483472 3873 ts_manager.cc:194] Registered new tserver with Master: 7de5d8e46d384f5481bcc9dcc10722af (127.0.198.3:33917)
I20260929 18:36:01.484062 3873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:55607
I20260929 18:36:01.489198 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[3828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[3828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:01.513996 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:01.505558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.0.198.62:44675)
0929 18:36:01.505717 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:01.505721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:01.505806 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:36:01.505965 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:01.505996 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:01.506052 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:01.506128 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:01.506829 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.507350 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.508167 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:01.508401 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:01.508495 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:01.511020 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:01.511023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:01.511025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:01.511051 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:01.512368 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.512771 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.512775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.512777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.512894 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:01.513132 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:01.513135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:01.513137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:01.513356 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:01.513472 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:01.513688 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:36:01.513806 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 18:36:01.514384 3873 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42612
I20260929 18:36:01.515149 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 3935
I20260929 18:36:01.519906 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4069
I20260929 18:36:01.524444 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4203
I20260929 18:36:01.529030 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 3841
2026-09-29T18:36:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2078 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[4344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[4344](info): set up 2 sockets
Sep 29 18:36:01 dist-test-slave-q5qx krb5kdc[4344](info): commencing operation
krb5kdc: starting...
W20260929 18:36:02.485874 792 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:02 dist-test-slave-q5qx krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:36:02.601245 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44641
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:02.709941 4357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:02.710245 4357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:02.710341 4357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:02.714490 4357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:02.714567 4357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:02.714594 4357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:02.714615 4357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:02.714641 4357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:02.720167 4357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:02.721256 4357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:02.722329 4357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:02.730530 4357 server_base.cc:1034] running on GCE node
W20260929 18:36:02.730541 4362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:02.730541 4363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:02.730722 4365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:02.731277 4357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:02.732380 4357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:02.733541 4357 hybrid_clock.cc:648] HybridClock initialized: now 1790706962733525 us; error 36 us; skew 500 ppm
Sep 29 18:36:02 dist-test-slave-q5qx krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:02.736150 4357 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:36:02.737067 4357 webserver.cc:460] Webserver started at http://127.0.198.62:36313/ using document root <none> and password file <none>
I20260929 18:36:02.737589 4357 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:02.737648 4357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:02.737824 4357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:02.739856 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "a22006bd23264b8eb30223dfe2fcc5ae"
format_stamp: "Formatted at 2026-09-29 18:36:02 on dist-test-slave-q5qx"
server_key: "c35be71fd38f8e80e8b02b511a7dfacf"
server_key_iv: "cbd6b1bd89e7623e3568b18c1473f785"
server_key_version: "encryptionkey@0"
I20260929 18:36:02.740382 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "a22006bd23264b8eb30223dfe2fcc5ae"
format_stamp: "Formatted at 2026-09-29 18:36:02 on dist-test-slave-q5qx"
server_key: "c35be71fd38f8e80e8b02b511a7dfacf"
server_key_iv: "cbd6b1bd89e7623e3568b18c1473f785"
server_key_version: "encryptionkey@0"
I20260929 18:36:02.744015 4357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 18:36:02.746464 4372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:02.747714 4357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:02.747821 4357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "a22006bd23264b8eb30223dfe2fcc5ae"
format_stamp: "Formatted at 2026-09-29 18:36:02 on dist-test-slave-q5qx"
server_key: "c35be71fd38f8e80e8b02b511a7dfacf"
server_key_iv: "cbd6b1bd89e7623e3568b18c1473f785"
server_key_version: "encryptionkey@0"
I20260929 18:36:02.747927 4357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:02.762302 4357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:02.765314 4357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:02.765488 4357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:02.772323 4357 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:44641
I20260929 18:36:02.772349 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:44641 every 8 connection(s)
I20260929 18:36:02.773247 4357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:02.776034 4425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:02.777274 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4357
I20260929 18:36:02.777374 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:02.777633 792 external_mini_cluster.cc:1428] Setting key e971cd35f9a5a4aac29a017b3057d0e5
I20260929 18:36:02.782052 4425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae: Bootstrap starting.
I20260929 18:36:02.784565 4425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:02.785485 4425 log.cc:824] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:36:02 dist-test-slave-q5qx krb5kdc[4344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:02.787652 4425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae: No bootstrap required, opened a new log
I20260929 18:36:02.791205 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:02.778920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.0.198.62:44641)
0929 18:36:02.779295 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:02.779311 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:02.779346 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:36:02.779940 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:02.779955 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:02.780280 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:02.780545 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:02.781396 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:02.782419 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:02.783533 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:02.783989 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:02.784101 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:02.786993 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:02.786998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:02.787013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:02.787039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:02.788600 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:02.789007 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:02.789012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:02.789016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:02.789121 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:02.789364 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:02.789367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:02.789369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:02.789741 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:02.789921 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:02.790198 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 18:36:02.790478 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260929 18:36:02.791136 4425 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44641 } }
I20260929 18:36:02.791374 4425 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:02.791446 4425 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22006bd23264b8eb30223dfe2fcc5ae, State: Initialized, Role: FOLLOWER
I20260929 18:36:02.791937 4425 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44641 } }
I20260929 18:36:02.792091 4425 raft_consensus.cc:388] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:02.792152 4425 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:02.792244 4425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:02.793088 4425 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44641 } }
I20260929 18:36:02.793428 4425 leader_election.cc:302] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a22006bd23264b8eb30223dfe2fcc5ae; no voters:
I20260929 18:36:02.793713 4425 leader_election.cc:288] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:02.793819 4430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:02.794035 4430 raft_consensus.cc:686] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [term 1 LEADER]: Becoming Leader. State: Replica: a22006bd23264b8eb30223dfe2fcc5ae, State: Running, Role: LEADER
I20260929 18:36:02.794365 4430 consensus_queue.cc:237] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44641 } }
I20260929 18:36:02.794857 4425 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:02.795892 4432 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22006bd23264b8eb30223dfe2fcc5ae. Latest consensus state: current_term: 1 leader_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44641 } } }
I20260929 18:36:02.795908 4431 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22006bd23264b8eb30223dfe2fcc5ae" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44641 } } }
I20260929 18:36:02.796020 4432 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:02.796039 4431 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:02.796377 4439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:02.799770 4439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:02.805434 4439 catalog_manager.cc:1347] Generated new cluster ID: 8500b2b49ae04f8a86a6213d0665077b
I20260929 18:36:02.805521 4439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:02.814391 4439 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:02.815467 4439 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:02.822559 4439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a22006bd23264b8eb30223dfe2fcc5ae: Generated new TSK 0
I20260929 18:36:02.823310 4439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:36:02.853258 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44641
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:02.961869 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:02.962147 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:02.962188 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:02.966107 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:02.966159 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:02.966215 4451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:02.971479 4451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:02.972455 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:02.973395 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:02.982770 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:02.982894 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:02.982980 4451 server_base.cc:1034] running on GCE node
W20260929 18:36:02.982890 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:02.983675 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:02.984265 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:02.985430 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790706962985405 us; error 39 us; skew 500 ppm
Sep 29 18:36:02 dist-test-slave-q5qx krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:02.988240 4451 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:36:02.989209 4451 webserver.cc:460] Webserver started at http://127.0.198.1:35755/ using document root <none> and password file <none>
I20260929 18:36:02.989750 4451 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:02.989816 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:02.990038 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:02.992051 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "b31c2e58b11d4b8097ae01f873744b84"
format_stamp: "Formatted at 2026-09-29 18:36:02 on dist-test-slave-q5qx"
server_key: "4f5d8ddea7c68c5634a489df3945e1a5"
server_key_iv: "d2f07be5f4d7c4f248798ff49d0e2a70"
server_key_version: "encryptionkey@0"
I20260929 18:36:02.992568 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "b31c2e58b11d4b8097ae01f873744b84"
format_stamp: "Formatted at 2026-09-29 18:36:02 on dist-test-slave-q5qx"
server_key: "4f5d8ddea7c68c5634a489df3945e1a5"
server_key_iv: "d2f07be5f4d7c4f248798ff49d0e2a70"
server_key_version: "encryptionkey@0"
I20260929 18:36:02.996279 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:36:02.998667 4466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:02.999886 4451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:36:03.000010 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "b31c2e58b11d4b8097ae01f873744b84"
format_stamp: "Formatted at 2026-09-29 18:36:02 on dist-test-slave-q5qx"
server_key: "4f5d8ddea7c68c5634a489df3945e1a5"
server_key_iv: "d2f07be5f4d7c4f248798ff49d0e2a70"
server_key_version: "encryptionkey@0"
I20260929 18:36:03.000109 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:03.008968 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:03.012024 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:03.012205 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:03.012778 4451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:03.013583 4451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:03.013629 4451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.013690 4451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:03.013718 4451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.022173 4451 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:35715
I20260929 18:36:03.022222 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:35715 every 8 connection(s)
I20260929 18:36:03.023051 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:03.029502 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4451
I20260929 18:36:03.029631 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:03.029924 792 external_mini_cluster.cc:1428] Setting key 6577a7f48deca67c1e8ea3f5136fcb8f
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:03.037137 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.024814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:42425 (local address 127.0.198.62:44641)
0929 18:36:03.024981 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:03.024985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:03.025799 (+ 814us) server_negotiation.cc:407] Connection header received
0929 18:36:03.027182 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:03.027186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:03.027240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:03.027312 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:03.028471 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.029001 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.030525 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.030766 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.030857 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.033436 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:03.033439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:03.033441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:03.033465 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:03.034954 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.035560 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.035564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.035566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.035655 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.035972 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.035976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.035977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.036218 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:03.036330 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:03.036815 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 18:36:03.036953 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":74}
I20260929 18:36:03.037879 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.025188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44641 (local address 127.0.198.1:42425)
0929 18:36:03.025631 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:03.025695 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:03.026974 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:03.027478 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:03.027486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:03.027838 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:03.028332 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:03.028346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.029118 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.029121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:03.030410 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:03.030421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.030907 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.030911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:03.031088 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.031643 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:03.031667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:03.033258 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:03.035074 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:03.035079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:03.035092 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:03.035457 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:03.035752 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:03.035754 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:03.035755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:03.035888 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:03.036323 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:03.036329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:03.036576 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 18:36:03.036744 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 18:36:03.037005 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:36:03.039376 4580 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44641
I20260929 18:36:03.039589 4580 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:03.040043 4580 heartbeater.cc:511] Master 127.0.198.62:44641 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:36:03.041502 4389 ts_manager.cc:194] Registered new tserver with Master: b31c2e58b11d4b8097ae01f873744b84 (127.0.198.1:35715)
I20260929 18:36:03.042690 4389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:42425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:36:03.056995 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44641
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:03.165364 4585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:03.165638 4585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:03.165681 4585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:03.169725 4585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:03.169798 4585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:03.169857 4585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:03.175204 4585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:03.176270 4585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:03.177263 4585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:03.186771 4590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:03.186771 4593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:03.186790 4591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:03.187098 4585 server_base.cc:1034] running on GCE node
I20260929 18:36:03.187517 4585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:03.188251 4585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:03.189411 4585 hybrid_clock.cc:648] HybridClock initialized: now 1790706963189383 us; error 45 us; skew 500 ppm
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:03.192754 4585 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:36:03.193598 4585 webserver.cc:460] Webserver started at http://127.0.198.2:43251/ using document root <none> and password file <none>
I20260929 18:36:03.194229 4585 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:03.194321 4585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:03.194587 4585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:03.196760 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "3806e6ca64684343989523dc510cec8e"
format_stamp: "Formatted at 2026-09-29 18:36:03 on dist-test-slave-q5qx"
server_key: "3e2f3ed89b9bfd5955dd2f934a9e47a8"
server_key_iv: "8f0de9689af1dbb14708548fae4633c7"
server_key_version: "encryptionkey@0"
I20260929 18:36:03.197248 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "3806e6ca64684343989523dc510cec8e"
format_stamp: "Formatted at 2026-09-29 18:36:03 on dist-test-slave-q5qx"
server_key: "3e2f3ed89b9bfd5955dd2f934a9e47a8"
server_key_iv: "8f0de9689af1dbb14708548fae4633c7"
server_key_version: "encryptionkey@0"
I20260929 18:36:03.200771 4585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 18:36:03.203249 4600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.204460 4585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:03.204555 4585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "3806e6ca64684343989523dc510cec8e"
format_stamp: "Formatted at 2026-09-29 18:36:03 on dist-test-slave-q5qx"
server_key: "3e2f3ed89b9bfd5955dd2f934a9e47a8"
server_key_iv: "8f0de9689af1dbb14708548fae4633c7"
server_key_version: "encryptionkey@0"
I20260929 18:36:03.204659 4585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:03.233297 4585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:03.236557 4585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:03.236748 4585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:03.237353 4585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:03.238257 4585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:03.238308 4585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.238361 4585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:03.238389 4585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.246995 4585 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42067
I20260929 18:36:03.247020 4713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42067 every 8 connection(s)
I20260929 18:36:03.247885 4585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:03.253285 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4585
I20260929 18:36:03.253391 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:03.253646 792 external_mini_cluster.cc:1428] Setting key 140514f2b1b1d7737ff705b960b46d82
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:03.262733 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.249815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:45979 (local address 127.0.198.62:44641)
0929 18:36:03.250002 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:03.250007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:03.250844 (+ 837us) server_negotiation.cc:407] Connection header received
0929 18:36:03.252352 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:03.252354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:03.252404 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:03.252489 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:03.253761 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.254515 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.255878 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.256183 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.256284 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.258697 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:03.258702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:03.258705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:03.258736 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:03.260414 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.260950 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.260954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.260956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.261048 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.261444 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.261448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.261450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.261720 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:03.261822 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:03.262417 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 18:36:03.262548 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:36:03.263624 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.250231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44641 (local address 127.0.198.2:45979)
0929 18:36:03.250683 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:03.250736 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:03.252135 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:03.252635 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:03.252642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:03.253006 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:03.253601 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:03.253616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.254653 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.254674 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:03.255754 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:03.255765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.256324 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.256327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:03.256475 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.257067 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:03.257083 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:03.258519 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:03.260534 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:03.260539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:03.260554 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:03.260857 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:03.261187 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:03.261190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:03.261192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:03.261343 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:03.261824 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:03.261829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:03.262152 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 18:36:03.262342 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 18:36:03.262620 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260929 18:36:03.264959 4714 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:36:03.265259 4714 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:03.265825 4714 heartbeater.cc:511] Master 127.0.198.62:44641 requested a full tablet report, sending...
I20260929 18:36:03.267071 4389 ts_manager.cc:194] Registered new tserver with Master: 3806e6ca64684343989523dc510cec8e (127.0.198.2:42067)
I20260929 18:36:03.267653 4389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:45979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:36:03.281548 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44641
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:03.394347 4719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:03.394788 4719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:03.394871 4719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:03.399380 4719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:03.399446 4719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:03.399533 4719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:03.404922 4719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:03.406136 4719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:03.407197 4719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:03.416723 4727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:03.416738 4725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:03.416724 4724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:03.416822 4719 server_base.cc:1034] running on GCE node
I20260929 18:36:03.417361 4719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:03.418010 4719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:03.419176 4719 hybrid_clock.cc:648] HybridClock initialized: now 1790706963419147 us; error 44 us; skew 500 ppm
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:03.421813 4719 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:36:03.422721 4719 webserver.cc:460] Webserver started at http://127.0.198.3:33897/ using document root <none> and password file <none>
I20260929 18:36:03.423230 4719 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:03.423288 4719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:03.423448 4719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:03.425432 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "66bf470f2fdc4e07aa621f8f266aed91"
format_stamp: "Formatted at 2026-09-29 18:36:03 on dist-test-slave-q5qx"
server_key: "5b2a293e6627518db500a112f5023cc4"
server_key_iv: "9ce2621755662cead7453642c1adea3c"
server_key_version: "encryptionkey@0"
I20260929 18:36:03.425920 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "66bf470f2fdc4e07aa621f8f266aed91"
format_stamp: "Formatted at 2026-09-29 18:36:03 on dist-test-slave-q5qx"
server_key: "5b2a293e6627518db500a112f5023cc4"
server_key_iv: "9ce2621755662cead7453642c1adea3c"
server_key_version: "encryptionkey@0"
I20260929 18:36:03.429538 4719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:03.432080 4734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.433554 4719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:03.433668 4719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "66bf470f2fdc4e07aa621f8f266aed91"
format_stamp: "Formatted at 2026-09-29 18:36:03 on dist-test-slave-q5qx"
server_key: "5b2a293e6627518db500a112f5023cc4"
server_key_iv: "9ce2621755662cead7453642c1adea3c"
server_key_version: "encryptionkey@0"
I20260929 18:36:03.433768 4719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:03.446969 4719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:03.450188 4719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:03.450344 4719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:03.450881 4719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:03.451855 4719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:03.451903 4719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.451942 4719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:03.451967 4719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:03.460522 4719 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:42473
I20260929 18:36:03.460625 4847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:42473 every 8 connection(s)
I20260929 18:36:03.461485 4719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:03.467871 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4719
I20260929 18:36:03.468007 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:03.468286 792 external_mini_cluster.cc:1428] Setting key 710003144c0d7ba79f2a8b38df2816ee
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:03.476261 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.463360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:44293 (local address 127.0.198.62:44641)
0929 18:36:03.463478 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:03.463482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:03.464448 (+ 966us) server_negotiation.cc:407] Connection header received
0929 18:36:03.465937 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:03.465941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:03.466057 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:03.466131 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:03.467530 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.468289 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.469751 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.470050 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.470133 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.472738 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:03.472741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:03.472743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:03.472770 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:03.474184 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.474721 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.474724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.474725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.474803 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.475219 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.475221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.475223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.475396 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:03.475462 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:03.476007 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 18:36:03.476116 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:36:03.477105 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.463783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44641 (local address 127.0.198.3:44293)
0929 18:36:03.464278 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:03.464322 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:03.465604 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:03.466330 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:03.466339 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:03.466723 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:03.467342 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:03.467359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.468439 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.468442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:03.469594 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:03.469614 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.470236 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.470240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:03.470444 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.471040 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:03.471057 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:03.472551 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:03.474305 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:03.474310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:03.474325 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:03.474612 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:03.474930 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:03.474933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:03.474934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:03.475137 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:03.475495 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:03.475502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:03.475762 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 18:36:03.475923 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 18:36:03.476247 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260929 18:36:03.478250 4848 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44641
I20260929 18:36:03.478488 4848 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:03.478993 4848 heartbeater.cc:511] Master 127.0.198.62:44641 requested a full tablet report, sending...
I20260929 18:36:03.480115 4389 ts_manager.cc:194] Registered new tserver with Master: 66bf470f2fdc4e07aa621f8f266aed91 (127.0.198.3:42473)
I20260929 18:36:03.480662 4389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:44293
I20260929 18:36:03.482445 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:36:03.508380 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.498258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.0.198.1:35715)
0929 18:36:03.498526 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:03.498541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:03.498563 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:36:03.498744 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:03.498749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:03.498980 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:03.499101 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:03.499850 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.500662 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.501661 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.502127 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.502224 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.504622 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:03.504631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:03.504645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:03.504686 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:03.506423 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.506857 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.506862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.506863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.506946 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.507311 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.507315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.507316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.507604 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:03.507796 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:03.508041 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:36:03.508182 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:03.519162 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:03.511426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.0.198.62:44641)
0929 18:36:03.511554 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:03.511560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:03.511598 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:36:03.511726 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:03.511729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:03.511773 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:03.511834 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:03.512413 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.512929 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.513682 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:03.513897 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:03.514033 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:03.516303 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:03.516306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:03.516309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:03.516334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:03.517576 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.517949 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.517952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.517954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.518068 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:03.518313 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:03.518316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:03.518318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:03.518545 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:03.518708 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:03.518891 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:36:03.519004 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 18:36:03.519526 4389 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42112
I20260929 18:36:03.520141 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4451
I20260929 18:36:03.525365 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4585
I20260929 18:36:03.530076 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4719
I20260929 18:36:03.534889 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4357
2026-09-29T18:36:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2005 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4860](info): set up 2 sockets
Sep 29 18:36:03 dist-test-slave-q5qx krb5kdc[4860](info): commencing operation
krb5kdc: starting...
W20260929 18:36:04.493930 792 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:04 dist-test-slave-q5qx krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:36:04.614240 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39687
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:04.724413 4873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:04.724687 4873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:04.724728 4873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:04.728830 4873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:04.728896 4873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:04.728914 4873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:04.728935 4873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:04.728950 4873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:04.734213 4873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:04.735244 4873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:04.736222 4873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:04.744247 4878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:04.744211 4881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:04.744302 4873 server_base.cc:1034] running on GCE node
W20260929 18:36:04.744231 4879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:04.744995 4873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:04.746138 4873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:04.747308 4873 hybrid_clock.cc:648] HybridClock initialized: now 1790706964747290 us; error 30 us; skew 500 ppm
Sep 29 18:36:04 dist-test-slave-q5qx krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:04.750187 4873 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:36:04.750924 4873 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:04.750995 4873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:04.751186 4873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:04.753265 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "cb5a87c9b79b483887c8430561fce801"
format_stamp: "Formatted at 2026-09-29 18:36:04 on dist-test-slave-q5qx"
server_key: "883fdeff6b3c45bd1794c2dbc908469d"
server_key_iv: "a760c602e52341949d3bc7beed19d969"
server_key_version: "encryptionkey@0"
I20260929 18:36:04.753822 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "cb5a87c9b79b483887c8430561fce801"
format_stamp: "Formatted at 2026-09-29 18:36:04 on dist-test-slave-q5qx"
server_key: "883fdeff6b3c45bd1794c2dbc908469d"
server_key_iv: "a760c602e52341949d3bc7beed19d969"
server_key_version: "encryptionkey@0"
I20260929 18:36:04.757748 4873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:36:04.760460 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:04.761752 4873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:04.761869 4873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "cb5a87c9b79b483887c8430561fce801"
format_stamp: "Formatted at 2026-09-29 18:36:04 on dist-test-slave-q5qx"
server_key: "883fdeff6b3c45bd1794c2dbc908469d"
server_key_iv: "a760c602e52341949d3bc7beed19d969"
server_key_version: "encryptionkey@0"
I20260929 18:36:04.762020 4873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:04.771481 4873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:04.774513 4873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:04.774698 4873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:04.781060 4873 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:39687
I20260929 18:36:04.781312 4939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:39687 every 8 connection(s)
I20260929 18:36:04.781710 4873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:04.784569 4940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:04.790263 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4873
I20260929 18:36:04.790376 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:04.790372 4940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801: Bootstrap starting.
I20260929 18:36:04.790627 792 external_mini_cluster.cc:1428] Setting key a215f4d541166f973dbee8f1e3226cb7
I20260929 18:36:04.793027 4940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:04.793906 4940 log.cc:824] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:04.796311 4940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801: No bootstrap required, opened a new log
Sep 29 18:36:04 dist-test-slave-q5qx krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:04.800705 4940 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5a87c9b79b483887c8430561fce801" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39687 } }
I20260929 18:36:04.800971 4940 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:04.801044 4940 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5a87c9b79b483887c8430561fce801, State: Initialized, Role: FOLLOWER
I20260929 18:36:04.801544 4940 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5a87c9b79b483887c8430561fce801" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39687 } }
I20260929 18:36:04.801716 4940 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:04.801792 4940 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:04.801900 4940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:04.802992 4940 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5a87c9b79b483887c8430561fce801" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39687 } }
I20260929 18:36:04.803344 4940 leader_election.cc:302] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5a87c9b79b483887c8430561fce801; no voters:
I20260929 18:36:04.803689 4940 leader_election.cc:288] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:04.803768 4945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:04.803969 4945 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [term 1 LEADER]: Becoming Leader. State: Replica: cb5a87c9b79b483887c8430561fce801, State: Running, Role: LEADER
I20260929 18:36:04.804337 4945 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5a87c9b79b483887c8430561fce801" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39687 } }
I20260929 18:36:04.805055 4940 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:04.805570 4945 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5a87c9b79b483887c8430561fce801. Latest consensus state: current_term: 1 leader_uuid: "cb5a87c9b79b483887c8430561fce801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5a87c9b79b483887c8430561fce801" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39687 } } }
I20260929 18:36:04.805684 4945 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:04.806162 4950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:04.806123 4946 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5a87c9b79b483887c8430561fce801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5a87c9b79b483887c8430561fce801" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39687 } } }
I20260929 18:36:04.806298 4946 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:04.806942 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:04.791967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.0.198.62:39687)
0929 18:36:04.792397 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:04.792416 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:04.792460 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:36:04.793319 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:04.793343 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:04.793769 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:04.794187 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:04.794983 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:04.796419 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:04.797388 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:04.798307 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:04.798480 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:04.801109 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:04.801116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:04.801135 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:04.801171 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:04.803039 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:04.803462 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:04.803477 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:04.803483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:04.803609 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:04.803906 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:04.803909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:04.803911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:04.804422 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:04.804618 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:04.805543 (+ 925us) server_negotiation.cc:299] Negotiation successful
0929 18:36:04.805877 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260929 18:36:04.810586 4950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:04.816529 4950 catalog_manager.cc:1347] Generated new cluster ID: 5e014efa5e824746b29d1d5972a73036
I20260929 18:36:04.816620 4950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:04.823167 4950 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:04.824451 4950 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:04.833129 4950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801: Generated new TSK 0
I20260929 18:36:04.834002 4950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:36:04.861840 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39687
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:04.974174 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:04.974462 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:04.974525 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:04.978554 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:04.978613 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:04.978695 4966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:04.983927 4966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:04.984954 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:04.985949 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:04.995132 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:04.995067 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:04.995182 4966 server_base.cc:1034] running on GCE node
W20260929 18:36:04.995046 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:04.995728 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:04.996349 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:04.997520 4966 hybrid_clock.cc:648] HybridClock initialized: now 1790706964997503 us; error 32 us; skew 500 ppm
Sep 29 18:36:04 dist-test-slave-q5qx krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:05.000254 4966 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:36:05.000965 4966 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:05.001032 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:05.001212 4966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:05.003230 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c"
format_stamp: "Formatted at 2026-09-29 18:36:04 on dist-test-slave-q5qx"
server_key: "653a24d9bdde2d0265948e98043ddcb8"
server_key_iv: "eb0d651199078ba4c4e5585dc46da86f"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.003757 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c"
format_stamp: "Formatted at 2026-09-29 18:36:04 on dist-test-slave-q5qx"
server_key: "653a24d9bdde2d0265948e98043ddcb8"
server_key_iv: "eb0d651199078ba4c4e5585dc46da86f"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.007728 4966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 18:36:05.010437 4980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.011842 4966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:05.011972 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c"
format_stamp: "Formatted at 2026-09-29 18:36:04 on dist-test-slave-q5qx"
server_key: "653a24d9bdde2d0265948e98043ddcb8"
server_key_iv: "eb0d651199078ba4c4e5585dc46da86f"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.012081 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:05.033849 4966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:05.037038 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:05.037230 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:05.037804 4966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:05.038684 4966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:05.038731 4966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.038782 4966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:05.038811 4966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.047356 4966 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:45423
I20260929 18:36:05.047401 5093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:45423 every 8 connection(s)
I20260929 18:36:05.047998 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:05.048676 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 4966
I20260929 18:36:05.048785 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:05.049049 792 external_mini_cluster.cc:1428] Setting key 4f100ef397f407284fbea4b22e17f692
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
Sep 29 18:36:05 dist-test-slave-q5qx krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:05.064163 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.049740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:35689 (local address 127.0.198.62:39687)
0929 18:36:05.050089 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.050094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.050914 (+ 820us) server_negotiation.cc:407] Connection header received
0929 18:36:05.052278 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.052282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.052334 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.052417 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:05.053530 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.054168 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.055379 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.055619 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.055705 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.060292 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:05.060297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:05.060299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:05.060328 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:05.061791 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.062374 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.062376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.062378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.062471 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.062869 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.062873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.062875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.063124 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:05.063215 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.063922 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.064039 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260929 18:36:05.065299 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.050229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39687 (local address 127.0.198.1:35689)
0929 18:36:05.050751 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.050805 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.052076 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.052560 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.052566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.052905 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:05.053389 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.053402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.054292 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.054294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.055267 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.055277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.055757 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.055761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.055937 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.056465 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:05.056481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:05.060109 (+ 3628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:05.061897 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:05.061903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:05.061921 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:05.062278 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:05.062557 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:05.062560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:05.062562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:05.062734 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:05.063238 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:05.063246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:05.063594 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.063808 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.064172 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":198,"threads_started":1}
I20260929 18:36:05.066707 5094 heartbeater.cc:347] Connected to a master server at 127.0.198.62:39687
I20260929 18:36:05.066989 5094 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:05.067428 5094 heartbeater.cc:511] Master 127.0.198.62:39687 requested a full tablet report, sending...
I20260929 18:36:05.068899 4902 ts_manager.cc:194] Registered new tserver with Master: 0bfea1f0b1f04d508451a0cd4a2ad35c (127.0.198.1:45423)
I20260929 18:36:05.070242 4902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:35689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:36:05.076948 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39687
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:05.184823 5099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:05.185117 5099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:05.185163 5099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:05.189242 5099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:05.189316 5099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:05.189378 5099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:05.194712 5099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:05.195691 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:05.196648 5099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:05.205834 5105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:05.205873 5107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:05.205929 5104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:05.206223 5099 server_base.cc:1034] running on GCE node
I20260929 18:36:05.206624 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:05.207283 5099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:05.208433 5099 hybrid_clock.cc:648] HybridClock initialized: now 1790706965208420 us; error 28 us; skew 500 ppm
Sep 29 18:36:05 dist-test-slave-q5qx krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:05.211112 5099 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:36:05.211910 5099 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:05.211977 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:05.212196 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:05.214277 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "a3f183a24a8c413ab609ea0e8c6db995"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "b89af896cacb1a97f8f1ee51fd49cb72"
server_key_iv: "a69d0e10afa94c641e1a3d966ea11fb4"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.214808 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "a3f183a24a8c413ab609ea0e8c6db995"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "b89af896cacb1a97f8f1ee51fd49cb72"
server_key_iv: "a69d0e10afa94c641e1a3d966ea11fb4"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.218664 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:05.221129 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.222471 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:05.222573 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "a3f183a24a8c413ab609ea0e8c6db995"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "b89af896cacb1a97f8f1ee51fd49cb72"
server_key_iv: "a69d0e10afa94c641e1a3d966ea11fb4"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.222667 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:05.235041 5099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:05.237754 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:05.237926 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:05.238533 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:05.239367 5099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:05.239423 5099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.239488 5099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:05.239513 5099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.247844 5099 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:41459
I20260929 18:36:05.247980 5226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:41459 every 8 connection(s)
I20260929 18:36:05.248490 5099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:05.252889 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 5099
I20260929 18:36:05.253006 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:05.253263 792 external_mini_cluster.cc:1428] Setting key 92b0d2bce0e130bdd2dbc47bd763e158
Sep 29 18:36:05 dist-test-slave-q5qx krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:36:05.262851 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.250161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:52541 (local address 127.0.198.62:39687)
0929 18:36:05.250311 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.250314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.251156 (+ 842us) server_negotiation.cc:407] Connection header received
0929 18:36:05.252514 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.252518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.252573 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.252645 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:05.253996 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.254614 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.255958 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.256198 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.256255 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.259007 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:05.259010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:05.259012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:05.259035 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:05.260522 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.261111 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.261114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.261116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.261202 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.261605 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.261609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.261611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.261854 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:05.261997 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.262553 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.262675 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:36:05.263733 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.250549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39687 (local address 127.0.198.2:52541)
0929 18:36:05.251004 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.251048 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.252297 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.252810 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.252817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.253213 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:05.253840 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.253859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.254726 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.254728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.255812 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.255830 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.256424 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.256426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.256561 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.257106 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:05.257125 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:05.258831 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:05.260653 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:05.260658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:05.260670 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:05.260994 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:05.261319 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:05.261322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:05.261324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:05.261467 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:05.262018 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:05.262025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:05.262275 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.262506 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.262861 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:36:05.264914 5227 heartbeater.cc:347] Connected to a master server at 127.0.198.62:39687
I20260929 18:36:05.265177 5227 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:05.265615 5227 heartbeater.cc:511] Master 127.0.198.62:39687 requested a full tablet report, sending...
I20260929 18:36:05.266727 4902 ts_manager.cc:194] Registered new tserver with Master: a3f183a24a8c413ab609ea0e8c6db995 (127.0.198.2:41459)
I20260929 18:36:05.267270 4902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:52541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:36:05.280584 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39687
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:05.388303 5232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:05.388675 5232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:05.388742 5232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:05.394191 5232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:05.394253 5232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:05.394315 5232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:05.399706 5232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:05.400729 5232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:05.401703 5232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:05.411299 5232 server_base.cc:1034] running on GCE node
W20260929 18:36:05.411269 5237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:05.411561 5240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:05.411561 5238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:05.412065 5232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:05.412719 5232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:05.413884 5232 hybrid_clock.cc:648] HybridClock initialized: now 1790706965413856 us; error 45 us; skew 500 ppm
Sep 29 18:36:05 dist-test-slave-q5qx krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:05.416834 5232 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:36:05.417564 5232 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:05.417624 5232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:05.417810 5232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:05.419891 5232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "9524805708534ce8afdeaf664d7754c6"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "c3c7a7888b67ac4c9eedae8792dcbc4a"
server_key_iv: "6b60ad2512aef2a2e5a32ddb3e801c61"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.420432 5232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "9524805708534ce8afdeaf664d7754c6"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "c3c7a7888b67ac4c9eedae8792dcbc4a"
server_key_iv: "6b60ad2512aef2a2e5a32ddb3e801c61"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.424082 5232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:36:05.426664 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.428040 5232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:05.428143 5232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "9524805708534ce8afdeaf664d7754c6"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "c3c7a7888b67ac4c9eedae8792dcbc4a"
server_key_iv: "6b60ad2512aef2a2e5a32ddb3e801c61"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.428247 5232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:05.460489 5232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:05.463538 5232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:05.463711 5232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:05.464283 5232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:05.465093 5232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:05.465138 5232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.465179 5232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:05.465207 5232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.473278 5232 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:37305
I20260929 18:36:05.473305 5359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:37305 every 8 connection(s)
I20260929 18:36:05.473858 5232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:05.476780 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 5232
I20260929 18:36:05.476868 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:05.477129 792 external_mini_cluster.cc:1428] Setting key e9ed8da2a14d8666b4c784adb8f69660
Sep 29 18:36:05 dist-test-slave-q5qx krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:05.487015 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.475491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:56211 (local address 127.0.198.62:39687)
0929 18:36:05.475654 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.475657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.476480 (+ 823us) server_negotiation.cc:407] Connection header received
0929 18:36:05.477925 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.477928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.477995 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.478065 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:05.479231 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.479738 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.480837 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.481050 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.481135 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.483571 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:05.483575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:05.483577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:05.483601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:05.485006 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.485526 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.485529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.485530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.485605 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.485920 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.485924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.485925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.486142 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:05.486228 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.486721 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.486850 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:36:05.487852 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.475836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39687 (local address 127.0.198.3:56211)
0929 18:36:05.476320 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.476365 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.477630 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.478247 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.478254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.478623 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:05.479099 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.479112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.479857 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.479860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.480723 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.480732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.481179 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.481183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.481348 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.481875 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:05.481891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:05.483406 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:05.485130 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:05.485136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:05.485152 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:05.485425 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:05.485687 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:05.485690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:05.485692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:05.485834 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:05.486237 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:05.486243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:05.486498 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.486669 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.486991 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260929 18:36:05.489060 5360 heartbeater.cc:347] Connected to a master server at 127.0.198.62:39687
I20260929 18:36:05.489310 5360 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:05.489781 5360 heartbeater.cc:511] Master 127.0.198.62:39687 requested a full tablet report, sending...
I20260929 18:36:05.490792 4902 ts_manager.cc:194] Registered new tserver with Master: 9524805708534ce8afdeaf664d7754c6 (127.0.198.3:37305)
I20260929 18:36:05.491299 4902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:56211
I20260929 18:36:05.502279 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:05.511446 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.504214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.0.198.62:39687)
0929 18:36:05.504382 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.504387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.504521 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:36:05.504655 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.504658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.504703 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.504790 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:05.505464 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.506060 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.506987 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.507197 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.507299 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.508523 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:05.508526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:05.508529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:05.508551 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:05.509685 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.510206 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.510209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.510211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.510311 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:05.510585 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:05.510587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:05.510588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:05.510823 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:05.510930 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.511192 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.511290 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:36:05.513952 4902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:05.517042 4902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:05.530906 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.525915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.0.198.2:41459)
0929 18:36:05.526409 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.526424 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.526444 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:36:05.526525 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.526531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.526702 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.526833 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:05.527171 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.527929 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.529499 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.530228 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.530329 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.530477 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.530648 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.530751 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":110,"threads_started":1}
I20260929 18:36:05.530901 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.525186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:37305 (local address 127.0.0.1:39020)
0929 18:36:05.525713 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.525727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.525815 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.526586 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.526592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.526613 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:05.526746 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.526752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.528412 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.528415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.529590 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.529598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.530487 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.530491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.530572 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.530591 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.530631 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.530756 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":66,"thread_start_us":114,"threads_started":1}
I20260929 18:36:05.531090 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.525073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:45423 (local address 127.0.0.1:51390)
0929 18:36:05.525536 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.525581 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.525702 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.527290 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.527294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.527311 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:05.527393 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.527398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.528731 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.528734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.529735 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.529744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.530742 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.530745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.530827 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.530848 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.530893 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.530965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260929 18:36:05.531137 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.525340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.0.198.3:37305)
0929 18:36:05.526097 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.526110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.526132 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:36:05.526204 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.526209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.526412 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.526594 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:05.526876 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.528280 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.529697 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.530362 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.530727 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.530866 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.530913 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.530994 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":166,"threads_started":1}
I20260929 18:36:05.531637 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.526229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.198.1:45423)
0929 18:36:05.526868 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.526874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.526894 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:36:05.526959 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.526964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.527125 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.527526 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:05.527656 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.528621 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.529851 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.530611 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.531240 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.531370 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.531415 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.531496 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":81,"threads_started":1}
I20260929 18:36:05.531764 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.525778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:41459 (local address 127.0.0.1:34756)
0929 18:36:05.526056 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.526068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.526148 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.526924 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.526927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.526944 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:05.527057 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.527061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.528047 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.528051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.529365 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.529376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.530344 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.530348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.530452 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.530496 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.530609 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.531649 (+ 1040us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":38,"thread_start_us":79,"threads_started":1}
I20260929 18:36:05.534262 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 3f796110a114404eb5690e22ae02b0bd (DEFAULT_TABLE table=test-table [id=7a41c181057b45418eba31b812f5edc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:05.534262 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 3f796110a114404eb5690e22ae02b0bd (DEFAULT_TABLE table=test-table [id=7a41c181057b45418eba31b812f5edc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:05.534271 5028 tablet_service.cc:1467] Processing CreateTablet for tablet 3f796110a114404eb5690e22ae02b0bd (DEFAULT_TABLE table=test-table [id=7a41c181057b45418eba31b812f5edc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:05.535276 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f796110a114404eb5690e22ae02b0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:05.535305 5028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f796110a114404eb5690e22ae02b0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:05.535310 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f796110a114404eb5690e22ae02b0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:05.540859 5377 tablet_bootstrap.cc:492] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Bootstrap starting.
I20260929 18:36:05.542243 5379 tablet_bootstrap.cc:492] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c: Bootstrap starting.
I20260929 18:36:05.544163 5379 tablet_bootstrap.cc:654] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:05.544163 5377 tablet_bootstrap.cc:654] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:05.544584 5378 tablet_bootstrap.cc:492] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Bootstrap starting.
I20260929 18:36:05.545415 5379 log.cc:824] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:05.545415 5377 log.cc:824] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:05.546566 5378 tablet_bootstrap.cc:654] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:05.547550 5377 tablet_bootstrap.cc:492] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: No bootstrap required, opened a new log
I20260929 18:36:05.547550 5379 tablet_bootstrap.cc:492] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c: No bootstrap required, opened a new log
I20260929 18:36:05.547593 5378 log.cc:824] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:05.547741 5379 ts_tablet_manager.cc:1397] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:36:05.547741 5377 ts_tablet_manager.cc:1397] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:36:05.549342 5378 tablet_bootstrap.cc:492] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: No bootstrap required, opened a new log
I20260929 18:36:05.549522 5378 ts_tablet_manager.cc:1397] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:36:05.551698 5377 raft_consensus.cc:348] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.551698 5379 raft_consensus.cc:348] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.551980 5377 raft_consensus.cc:374] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:05.551980 5379 raft_consensus.cc:374] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:05.552029 5377 raft_consensus.cc:729] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f183a24a8c413ab609ea0e8c6db995, State: Initialized, Role: FOLLOWER
I20260929 18:36:05.552038 5379 raft_consensus.cc:729] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bfea1f0b1f04d508451a0cd4a2ad35c, State: Initialized, Role: FOLLOWER
I20260929 18:36:05.552378 5378 raft_consensus.cc:348] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.552532 5377 consensus_queue.cc:260] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.552532 5379 consensus_queue.cc:260] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.552628 5378 raft_consensus.cc:374] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:05.552675 5378 raft_consensus.cc:729] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9524805708534ce8afdeaf664d7754c6, State: Initialized, Role: FOLLOWER
I20260929 18:36:05.553109 5378 consensus_queue.cc:260] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.553541 5379 ts_tablet_manager.cc:1428] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:36:05.553550 5377 ts_tablet_manager.cc:1428] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:36:05.553807 5227 heartbeater.cc:503] Master 127.0.198.62:39687 was elected leader, sending a full tablet report...
I20260929 18:36:05.554062 5378 ts_tablet_manager.cc:1428] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:05.554281 5094 heartbeater.cc:503] Master 127.0.198.62:39687 was elected leader, sending a full tablet report...
I20260929 18:36:05.554363 5360 heartbeater.cc:503] Master 127.0.198.62:39687 was elected leader, sending a full tablet report...
I20260929 18:36:05.607712 5383 raft_consensus.cc:482] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:05.607930 5383 raft_consensus.cc:504] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.609015 5383 leader_election.cc:288] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9524805708534ce8afdeaf664d7754c6 (127.0.198.3:37305), a3f183a24a8c413ab609ea0e8c6db995 (127.0.198.2:41459)
I20260929 18:36:05.612634 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.609306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:40305 (local address 127.0.198.3:37305)
0929 18:36:05.609429 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.609433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.609570 (+ 137us) server_negotiation.cc:407] Connection header received
0929 18:36:05.609760 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.609764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.609819 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.609939 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:05.610210 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.610691 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.611749 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.612142 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.612231 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.612292 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.612453 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.612517 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:36:05.612651 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.609228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:37305 (local address 127.0.198.1:40305)
0929 18:36:05.609497 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.609509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.609607 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.609960 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.609962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.610004 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:05.610115 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.610120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.610794 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.610797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.611642 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.611651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.612248 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.612251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.612341 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.612363 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.612433 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.612506 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":20,"thread_start_us":113,"threads_started":1}
I20260929 18:36:05.612788 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.609760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:47479 (local address 127.0.198.2:41459)
0929 18:36:05.609874 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.609878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.609924 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:36:05.610074 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.610077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.610122 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.610179 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:05.610412 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.610885 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.611892 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.612283 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.612368 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.612422 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.612604 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.612667 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:36:05.612982 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.609628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:41459 (local address 127.0.198.1:47479)
0929 18:36:05.609864 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.609876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:05.609990 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:05.610195 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:05.610197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:05.610211 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:05.610312 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.610318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.610984 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.610988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.611779 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:05.611787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.612382 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.612385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:05.612493 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.612515 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:36:05.612578 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:36:05.612642 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":21,"thread_start_us":103,"threads_started":1}
I20260929 18:36:05.613268 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f796110a114404eb5690e22ae02b0bd" candidate_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9524805708534ce8afdeaf664d7754c6" is_pre_election: true
I20260929 18:36:05.613561 5313 raft_consensus.cc:2454] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bfea1f0b1f04d508451a0cd4a2ad35c in term 0.
I20260929 18:36:05.613550 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f796110a114404eb5690e22ae02b0bd" candidate_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f183a24a8c413ab609ea0e8c6db995" is_pre_election: true
I20260929 18:36:05.613792 5181 raft_consensus.cc:2454] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bfea1f0b1f04d508451a0cd4a2ad35c in term 0.
I20260929 18:36:05.614080 4984 leader_election.cc:302] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfea1f0b1f04d508451a0cd4a2ad35c, 9524805708534ce8afdeaf664d7754c6; no voters:
I20260929 18:36:05.614302 5383 raft_consensus.cc:2781] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:05.614379 5383 raft_consensus.cc:482] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:05.614408 5383 raft_consensus.cc:3037] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:05.615372 5383 raft_consensus.cc:504] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.615698 5383 leader_election.cc:288] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [CANDIDATE]: Term 1 election: Requested vote from peers 9524805708534ce8afdeaf664d7754c6 (127.0.198.3:37305), a3f183a24a8c413ab609ea0e8c6db995 (127.0.198.2:41459)
I20260929 18:36:05.616017 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f796110a114404eb5690e22ae02b0bd" candidate_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9524805708534ce8afdeaf664d7754c6"
I20260929 18:36:05.616068 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f796110a114404eb5690e22ae02b0bd" candidate_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f183a24a8c413ab609ea0e8c6db995"
I20260929 18:36:05.616125 5313 raft_consensus.cc:3037] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:05.616180 5181 raft_consensus.cc:3037] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:05.617166 5313 raft_consensus.cc:2454] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bfea1f0b1f04d508451a0cd4a2ad35c in term 1.
I20260929 18:36:05.617226 5181 raft_consensus.cc:2454] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bfea1f0b1f04d508451a0cd4a2ad35c in term 1.
I20260929 18:36:05.617517 4984 leader_election.cc:302] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfea1f0b1f04d508451a0cd4a2ad35c, a3f183a24a8c413ab609ea0e8c6db995; no voters:
I20260929 18:36:05.617662 5383 raft_consensus.cc:2781] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:05.617862 5383 raft_consensus.cc:686] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 1 LEADER]: Becoming Leader. State: Replica: 0bfea1f0b1f04d508451a0cd4a2ad35c, State: Running, Role: LEADER
I20260929 18:36:05.618227 5383 consensus_queue.cc:237] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } }
I20260929 18:36:05.621161 4901 catalog_manager.cc:5495] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bfea1f0b1f04d508451a0cd4a2ad35c (127.0.198.1). New cstate: current_term: 1 leader_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfea1f0b1f04d508451a0cd4a2ad35c" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 } health_report { overall_health: UNKNOWN } } }
I20260929 18:36:05.635190 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.631687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.0.198.1:45423)
0929 18:36:05.631807 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.631811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.631824 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:05.632008 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.632011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.632065 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.632150 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:05.632407 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.632932 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.633788 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:05.634044 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:05.634126 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:05.634430 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:05.634827 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:05.634939 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.634999 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.635054 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:36:05.640751 5313 raft_consensus.cc:1261] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 1 FOLLOWER]: Refusing update from remote peer 0bfea1f0b1f04d508451a0cd4a2ad35c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:05.640792 5181 raft_consensus.cc:1261] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 1 FOLLOWER]: Refusing update from remote peer 0bfea1f0b1f04d508451a0cd4a2ad35c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:05.641376 5383 consensus_queue.cc:1035] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9524805708534ce8afdeaf664d7754c6" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 37305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:05.641548 5388 consensus_queue.cc:1035] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f183a24a8c413ab609ea0e8c6db995" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:05.648947 5392 mvcc.cc:205] Tried to move back new op lower bound from 7334735731258142720 to 7334735731174719488. Current Snapshot: MvccSnapshot[applied={T|T < 7334735731258142720}]
I20260929 18:36:05.656807 4904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:05.657015 4904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:05.659500 4904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb5a87c9b79b483887c8430561fce801: Sending DeleteTablet for 3 replicas of tablet 3f796110a114404eb5690e22ae02b0bd
I20260929 18:36:05.660243 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f796110a114404eb5690e22ae02b0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:05 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:39020
I20260929 18:36:05.660265 5028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f796110a114404eb5690e22ae02b0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:05 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:51390
I20260929 18:36:05.660284 5161 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f796110a114404eb5690e22ae02b0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:05 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:34756
I20260929 18:36:05.660763 5401 tablet_replica.cc:330] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c: stopping tablet replica
I20260929 18:36:05.661124 5401 raft_consensus.cc:2229] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:05.661535 5401 raft_consensus.cc:2258] T 3f796110a114404eb5690e22ae02b0bd P 0bfea1f0b1f04d508451a0cd4a2ad35c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:05.661839 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4966
I20260929 18:36:05.663254 5403 tablet_replica.cc:330] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: stopping tablet replica
I20260929 18:36:05.663524 5403 raft_consensus.cc:2229] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:05.663800 5403 raft_consensus.cc:2258] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:05.664119 5402 tablet_replica.cc:330] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: stopping tablet replica
I20260929 18:36:05.664402 5402 raft_consensus.cc:2229] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:05.664659 5402 raft_consensus.cc:2258] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:05.665395 5403 ts_tablet_manager.cc:1905] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:05.665879 5402 ts_tablet_manager.cc:1905] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:05.668031 5402 ts_tablet_manager.cc:1918] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:05.668124 5402 log.cc:1196] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/3f796110a114404eb5690e22ae02b0bd
I20260929 18:36:05.668164 5403 ts_tablet_manager.cc:1918] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:05.668259 5403 log.cc:1196] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/3f796110a114404eb5690e22ae02b0bd
I20260929 18:36:05.668413 5402 ts_tablet_manager.cc:1939] T 3f796110a114404eb5690e22ae02b0bd P 9524805708534ce8afdeaf664d7754c6: Deleting consensus metadata
W20260929 18:36:05.668396 4889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:45423 (error 108)
I20260929 18:36:05.668592 5403 ts_tablet_manager.cc:1939] T 3f796110a114404eb5690e22ae02b0bd P a3f183a24a8c413ab609ea0e8c6db995: Deleting consensus metadata
I20260929 18:36:05.669441 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.669151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:45423 (local address 127.0.0.1:36654)
0929 18:36:05.669283 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:05.669355 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:36:05.669485 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 5099
I20260929 18:36:05.670022 4891 catalog_manager.cc:4841] TS a3f183a24a8c413ab609ea0e8c6db995 (127.0.198.2:41459): tablet 3f796110a114404eb5690e22ae02b0bd (table test-table [id=7a41c181057b45418eba31b812f5edc1]) successfully deleted
I20260929 18:36:05.672125 4891 catalog_manager.cc:4841] TS 9524805708534ce8afdeaf664d7754c6 (127.0.198.3:37305): tablet 3f796110a114404eb5690e22ae02b0bd (table test-table [id=7a41c181057b45418eba31b812f5edc1]) successfully deleted
W20260929 18:36:05.672189 4889 catalog_manager.cc:4584] TS 0bfea1f0b1f04d508451a0cd4a2ad35c (127.0.198.1:45423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f796110a114404eb5690e22ae02b0bd: Network error: Client connection negotiation failed: client connection to 127.0.198.1:45423: connect: Connection refused (error 111)
I20260929 18:36:05.675644 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 5232
I20260929 18:36:05.680083 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 4873
2026-09-29T18:36:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2145 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T18:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:05Z Disabled control of system clock
I20260929 18:36:05.707616 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:46859
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:36:05.813885 5408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:05.814184 5408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:05.814219 5408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:05.818302 5408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:05.818368 5408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:05.818383 5408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:05.818398 5408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:05.818411 5408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:05.823323 5408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:46859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:05.824393 5408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:05.825760 5408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:05.834179 5408 server_base.cc:1034] running on GCE node
W20260929 18:36:05.834132 5416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:05.834146 5413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:05.834139 5414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:05.835038 5408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:05.836169 5408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:05.837337 5408 hybrid_clock.cc:648] HybridClock initialized: now 1790706965837318 us; error 32 us; skew 500 ppm
I20260929 18:36:05.839192 5408 webserver.cc:460] Webserver started at http://127.0.198.62:37521/ using document root <none> and password file <none>
I20260929 18:36:05.839762 5408 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:05.839828 5408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:05.840018 5408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:05.842087 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "bc3cdcf8f7f848b78f4049274061e2e7"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "d2762a45dd611426bdb022258cde8044"
server_key_iv: "b9ee772ab7a455e310f1aac4e337a0f3"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.842667 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "bc3cdcf8f7f848b78f4049274061e2e7"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "d2762a45dd611426bdb022258cde8044"
server_key_iv: "b9ee772ab7a455e310f1aac4e337a0f3"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.846276 5408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:05.848683 5422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:05.849910 5408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:05.850055 5408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "bc3cdcf8f7f848b78f4049274061e2e7"
format_stamp: "Formatted at 2026-09-29 18:36:05 on dist-test-slave-q5qx"
server_key: "d2762a45dd611426bdb022258cde8044"
server_key_iv: "b9ee772ab7a455e310f1aac4e337a0f3"
server_key_version: "encryptionkey@0"
I20260929 18:36:05.850175 5408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:05.854591 5408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:05.855165 5408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:05.855310 5408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:05.861769 5408 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:46859
I20260929 18:36:05.861824 5474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:46859 every 8 connection(s)
I20260929 18:36:05.862699 5408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:05.865799 5475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:05.871322 5475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7: Bootstrap starting.
I20260929 18:36:05.872879 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 5408
I20260929 18:36:05.872973 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:05.873220 792 external_mini_cluster.cc:1428] Setting key f85c006ff74b3e0c979a080fa6f4aa6e
I20260929 18:36:05.873822 5475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:05.874701 5475 log.cc:824] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:05.876585 5475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7: No bootstrap required, opened a new log
I20260929 18:36:05.878312 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:05.874342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.0.198.62:46859)
0929 18:36:05.875002 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:05.875019 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:05.875055 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:36:05.875684 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:05.875701 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:05.875987 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:05.876290 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:05.877065 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:05.877070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:05.877082 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:05.877091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:05.877144 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:05.877159 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:05.877252 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:05.877369 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 18:36:05.877605 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":126,"threads_started":1}
I20260929 18:36:05.880055 5475 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46859 } }
I20260929 18:36:05.880309 5475 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:05.880366 5475 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3cdcf8f7f848b78f4049274061e2e7, State: Initialized, Role: FOLLOWER
I20260929 18:36:05.880716 5475 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46859 } }
I20260929 18:36:05.880833 5475 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:05.880875 5475 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:05.880941 5475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:05.881789 5475 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46859 } }
I20260929 18:36:05.882104 5475 leader_election.cc:302] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc3cdcf8f7f848b78f4049274061e2e7; no voters:
I20260929 18:36:05.882354 5475 leader_election.cc:288] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:05.882462 5480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:05.882670 5480 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [term 1 LEADER]: Becoming Leader. State: Replica: bc3cdcf8f7f848b78f4049274061e2e7, State: Running, Role: LEADER
I20260929 18:36:05.882977 5480 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46859 } }
I20260929 18:36:05.883589 5475 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:05.884517 5482 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46859 } } }
I20260929 18:36:05.884639 5482 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:05.884874 5481 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3cdcf8f7f848b78f4049274061e2e7. Latest consensus state: current_term: 1 leader_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3cdcf8f7f848b78f4049274061e2e7" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46859 } } }
I20260929 18:36:05.884958 5481 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:05.885156 5489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:05.887775 5489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:05.893356 5489 catalog_manager.cc:1347] Generated new cluster ID: 39b02ce1b6b04f4bb2422fdedf6748b0
I20260929 18:36:05.893416 5489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:05.911430 5489 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:05.911746 5489 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:05.918489 5489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7: Generated new TSK 0
I20260929 18:36:05.919286 5489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:05.925060 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46859
--builtin_ntp_servers=127.0.198.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:36:06.036966 5499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:06.037264 5499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:06.037310 5499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:06.041496 5499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:06.041560 5499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:06.041626 5499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:06.046496 5499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:06.047537 5499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:06.048907 5499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:06.058705 5504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.058709 5505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.058777 5499 server_base.cc:1034] running on GCE node
W20260929 18:36:06.058737 5507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.059376 5499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:06.060009 5499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:06.061228 5499 hybrid_clock.cc:648] HybridClock initialized: now 1790706966061209 us; error 31 us; skew 500 ppm
I20260929 18:36:06.063097 5499 webserver.cc:460] Webserver started at http://127.0.198.1:33633/ using document root <none> and password file <none>
I20260929 18:36:06.063665 5499 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:06.063732 5499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:06.063918 5499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:06.066075 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "eb02c1133e2c496bbe5df815cc3c2acd"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "a2b6f38d681c138dd6bfc6ad10b78257"
server_key_iv: "12660cfefebefdb280526d2c24c6d307"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.066622 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "eb02c1133e2c496bbe5df815cc3c2acd"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "a2b6f38d681c138dd6bfc6ad10b78257"
server_key_iv: "12660cfefebefdb280526d2c24c6d307"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.070462 5499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:36:06.072947 5513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.074306 5499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:06.074429 5499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "eb02c1133e2c496bbe5df815cc3c2acd"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "a2b6f38d681c138dd6bfc6ad10b78257"
server_key_iv: "12660cfefebefdb280526d2c24c6d307"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.074533 5499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:06.079197 5499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:06.079823 5499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:06.079985 5499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:06.080538 5499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:06.081455 5499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:06.081502 5499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.081558 5499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:06.081585 5499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.090003 5499 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:38213
I20260929 18:36:06.090029 5626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:38213 every 8 connection(s)
I20260929 18:36:06.090914 5499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:06.091147 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 5499
I20260929 18:36:06.091251 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:06.091566 792 external_mini_cluster.cc:1428] Setting key 889cd9a7423639a7fc95ec873a9da87d
I20260929 18:36:06.093591 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46859
--builtin_ntp_servers=127.0.198.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:36:06.097134 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.092681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:56813 (local address 127.0.198.62:46859)
0929 18:36:06.092816 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.092821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.094218 (+ 1397us) server_negotiation.cc:407] Connection header received
0929 18:36:06.095483 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.095486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.095531 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.095603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.096709 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.096713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.096716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.096730 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.096755 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.096766 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.096872 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.096954 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.096997 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:36:06.097958 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.093223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:46859 (local address 127.0.198.1:56813)
0929 18:36:06.094067 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.094111 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.095281 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.095772 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.095779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.095868 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.096417 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.096419 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.096461 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.096530 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.096852 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.096857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.096861 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.096985 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.097174 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260929 18:36:06.099262 5627 heartbeater.cc:347] Connected to a master server at 127.0.198.62:46859
I20260929 18:36:06.099493 5627 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:06.099884 5627 heartbeater.cc:511] Master 127.0.198.62:46859 requested a full tablet report, sending...
I20260929 18:36:06.101397 5439 ts_manager.cc:194] Registered new tserver with Master: eb02c1133e2c496bbe5df815cc3c2acd (127.0.198.1:38213)
W20260929 18:36:06.202050 5630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:06.202315 5630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:06.202355 5630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:06.206739 5630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:06.206795 5630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:06.206857 5630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:06.211642 5630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:06.212760 5630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:06.214178 5630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:06.223779 5630 server_base.cc:1034] running on GCE node
W20260929 18:36:06.223729 5635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.223691 5638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.223697 5636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.224365 5630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:06.225003 5630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:06.226181 5630 hybrid_clock.cc:648] HybridClock initialized: now 1790706966226162 us; error 27 us; skew 500 ppm
I20260929 18:36:06.233225 5630 webserver.cc:460] Webserver started at http://127.0.198.2:44661/ using document root <none> and password file <none>
I20260929 18:36:06.233819 5630 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:06.233893 5630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:06.234138 5630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:06.236222 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "60ec5f28bbb54f1e9ecae9e207ca6767"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "2e2446a3e23eb7437f010b7ca3e2ea20"
server_key_iv: "51ed11e6304920c292d4115fccb4d880"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.236759 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "60ec5f28bbb54f1e9ecae9e207ca6767"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "2e2446a3e23eb7437f010b7ca3e2ea20"
server_key_iv: "51ed11e6304920c292d4115fccb4d880"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.240566 5630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:06.243129 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.244369 5630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:06.244484 5630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "60ec5f28bbb54f1e9ecae9e207ca6767"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "2e2446a3e23eb7437f010b7ca3e2ea20"
server_key_iv: "51ed11e6304920c292d4115fccb4d880"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.244581 5630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:06.249104 5630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:06.249728 5630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:06.249881 5630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:06.250438 5630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:06.251330 5630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:06.251375 5630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.251425 5630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:06.251451 5630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.259665 5630 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:36467
I20260929 18:36:06.259742 5757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:36467 every 8 connection(s)
I20260929 18:36:06.260561 5630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:06.266721 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.262415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:33533 (local address 127.0.198.62:46859)
0929 18:36:06.262572 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.262577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.263649 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 18:36:06.264956 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.264959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.264999 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.265077 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.266240 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.266243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.266245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.266257 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.266278 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.266286 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.266369 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.266566 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.266610 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:36:06.267555 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.262810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:46859 (local address 127.0.198.2:33533)
0929 18:36:06.263500 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.263541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.264671 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.265281 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.265288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.265378 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.265930 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.265932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.265996 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.266085 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.266390 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.266398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.266403 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.266556 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.266752 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260929 18:36:06.268622 5758 heartbeater.cc:347] Connected to a master server at 127.0.198.62:46859
I20260929 18:36:06.268821 5758 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:06.269179 5758 heartbeater.cc:511] Master 127.0.198.62:46859 requested a full tablet report, sending...
I20260929 18:36:06.269588 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 5630
I20260929 18:36:06.269701 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:06.269956 792 external_mini_cluster.cc:1428] Setting key 040e6c89c8149d69552b215689c8c00a
I20260929 18:36:06.270146 5439 ts_manager.cc:194] Registered new tserver with Master: 60ec5f28bbb54f1e9ecae9e207ca6767 (127.0.198.2:36467)
I20260929 18:36:06.271895 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46859
--builtin_ntp_servers=127.0.198.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:36:06.382087 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:06.382375 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:06.382433 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:06.386811 5761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:06.386879 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:06.386956 5761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:06.392237 5761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:06.393370 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:06.394789 5761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:06.404562 5769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.404544 5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.404543 5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.405020 5761 server_base.cc:1034] running on GCE node
I20260929 18:36:06.405431 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:06.406114 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:06.407287 5761 hybrid_clock.cc:648] HybridClock initialized: now 1790706966407245 us; error 54 us; skew 500 ppm
I20260929 18:36:06.409096 5761 webserver.cc:460] Webserver started at http://127.0.198.3:42389/ using document root <none> and password file <none>
I20260929 18:36:06.409657 5761 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:06.409716 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:06.409910 5761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:06.412005 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "26d667db46004a76aaa5df0ed53eca1c"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "ec5265a4e953fbab70f5824ed71732a7"
server_key_iv: "e7dd02e8213c611c705530febb48a3d9"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.412559 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "26d667db46004a76aaa5df0ed53eca1c"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "ec5265a4e953fbab70f5824ed71732a7"
server_key_iv: "e7dd02e8213c611c705530febb48a3d9"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.416415 5761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 18:36:06.418856 5775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.420128 5761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 18:36:06.420231 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "26d667db46004a76aaa5df0ed53eca1c"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "ec5265a4e953fbab70f5824ed71732a7"
server_key_iv: "e7dd02e8213c611c705530febb48a3d9"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.420344 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:06.424753 5761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:06.425329 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:06.425477 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:06.426040 5761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:06.427003 5761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:06.427052 5761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.427090 5761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:06.427117 5761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.435485 5761 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:32799
I20260929 18:36:06.435520 5888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:32799 every 8 connection(s)
I20260929 18:36:06.436374 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:06.437960 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 5761
I20260929 18:36:06.438100 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:06.438391 792 external_mini_cluster.cc:1428] Setting key c6784f8ec379d1815adfa864fd3d188d
I20260929 18:36:06.442548 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.438255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:40645 (local address 127.0.198.62:46859)
0929 18:36:06.438385 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.438389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.439667 (+ 1278us) server_negotiation.cc:407] Connection header received
0929 18:36:06.441010 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.441013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.441050 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.441115 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.442164 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.442167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.442170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.442181 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.442200 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.442208 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.442300 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.442398 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.442443 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:36:06.443476 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.438791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:46859 (local address 127.0.198.3:40645)
0929 18:36:06.439518 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.439569 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.440818 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.441252 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.441259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.441349 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.441873 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.441875 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.441920 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.442023 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.442293 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.442298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.442302 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.442430 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.442634 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
I20260929 18:36:06.444628 5889 heartbeater.cc:347] Connected to a master server at 127.0.198.62:46859
I20260929 18:36:06.444854 5889 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:06.445235 5889 heartbeater.cc:511] Master 127.0.198.62:46859 requested a full tablet report, sending...
I20260929 18:36:06.446249 5439 ts_manager.cc:194] Registered new tserver with Master: 26d667db46004a76aaa5df0ed53eca1c (127.0.198.3:32799)
I20260929 18:36:06.452433 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:06.455950 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.454406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.0.198.62:46859)
0929 18:36:06.454522 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.454527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.454700 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:36:06.454823 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.454826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.454871 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.454936 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.455582 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.455585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.455588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.455599 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.455620 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.455628 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.455698 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.455764 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.455804 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:36:06.459033 5439 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:06.462034 5439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:06.474345 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.471895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:36467 (local address 127.0.0.1:37474)
0929 18:36:06.472194 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.472206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.472263 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.473062 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.473065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.473080 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.473498 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.473500 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.473521 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.473593 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.474045 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.474048 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.474052 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.474107 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.474180 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
I20260929 18:36:06.474397 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.471178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.0.198.3:32799)
0929 18:36:06.472362 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.472374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.472397 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:36:06.472449 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.472454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.472620 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.472742 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.473894 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.473899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.473907 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.473919 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.473955 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.474013 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.474107 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.474156 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.474227 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":127,"threads_started":1}
I20260929 18:36:06.474395 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.471337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.0.198.1:38213)
0929 18:36:06.471756 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.471772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.471973 (+ 201us) server_negotiation.cc:407] Connection header received
0929 18:36:06.472142 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.472148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.472386 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.472503 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.473769 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.473776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.473789 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.473801 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.473849 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.473882 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.474019 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.474160 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.474231 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":111,"threads_started":1}
I20260929 18:36:06.474346 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.471087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:32799 (local address 127.0.0.1:47220)
0929 18:36:06.471900 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.471910 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.472007 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.472767 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.472773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.472851 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.473742 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.473744 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.473753 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.473775 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.474077 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.474080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.474083 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.474147 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.474183 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":450,"thread_start_us":101,"threads_started":1}
I20260929 18:36:06.475266 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.472276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.198.2:36467)
0929 18:36:06.472658 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.472668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.472690 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:36:06.472737 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.472741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.472914 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.473286 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.473816 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.473821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.473832 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.473841 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.473889 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.473916 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.474979 (+ 1063us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.475030 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.475099 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260929 18:36:06.475612 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.471087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38213 (local address 127.0.0.1:34580)
0929 18:36:06.471821 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.471875 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.472020 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.473208 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.473211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.473225 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.473608 (+ 383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.473610 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.473621 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.473648 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.473961 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.474002 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.474008 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.475463 (+ 1455us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.475503 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":96,"threads_started":1}
I20260929 18:36:06.477629 5823 tablet_service.cc:1467] Processing CreateTablet for tablet 14ca7af7180541208d0bddc076cce077 (DEFAULT_TABLE table=test-table [id=d8239850740b47f0b832d04c8d376024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:06.478006 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 14ca7af7180541208d0bddc076cce077 (DEFAULT_TABLE table=test-table [id=d8239850740b47f0b832d04c8d376024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:06.478255 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 14ca7af7180541208d0bddc076cce077 (DEFAULT_TABLE table=test-table [id=d8239850740b47f0b832d04c8d376024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:06.478888 5823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ca7af7180541208d0bddc076cce077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:06.479056 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ca7af7180541208d0bddc076cce077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:06.479164 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ca7af7180541208d0bddc076cce077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:06.485853 5906 tablet_bootstrap.cc:492] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd: Bootstrap starting.
I20260929 18:36:06.486726 5907 tablet_bootstrap.cc:492] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Bootstrap starting.
I20260929 18:36:06.488302 5906 tablet_bootstrap.cc:654] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:06.488680 5907 tablet_bootstrap.cc:654] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:06.489027 5906 log.cc:824] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:06.489313 5907 log.cc:824] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:06.489935 5908 tablet_bootstrap.cc:492] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Bootstrap starting.
I20260929 18:36:06.491000 5907 tablet_bootstrap.cc:492] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: No bootstrap required, opened a new log
I20260929 18:36:06.491091 5906 tablet_bootstrap.cc:492] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd: No bootstrap required, opened a new log
I20260929 18:36:06.491180 5907 ts_tablet_manager.cc:1397] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:36:06.491266 5906 ts_tablet_manager.cc:1397] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:36:06.492089 5908 tablet_bootstrap.cc:654] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:06.492872 5908 log.cc:824] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:06.494840 5907 raft_consensus.cc:348] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.495056 5908 tablet_bootstrap.cc:492] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: No bootstrap required, opened a new log
I20260929 18:36:06.495064 5907 raft_consensus.cc:374] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:06.495111 5907 raft_consensus.cc:729] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ec5f28bbb54f1e9ecae9e207ca6767, State: Initialized, Role: FOLLOWER
I20260929 18:36:06.495257 5908 ts_tablet_manager.cc:1397] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:36:06.495543 5907 consensus_queue.cc:260] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.495592 5906 raft_consensus.cc:348] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.495846 5906 raft_consensus.cc:374] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:06.495906 5906 raft_consensus.cc:729] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb02c1133e2c496bbe5df815cc3c2acd, State: Initialized, Role: FOLLOWER
I20260929 18:36:06.496388 5906 consensus_queue.cc:260] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.496470 5907 ts_tablet_manager.cc:1428] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:36:06.497009 5627 heartbeater.cc:503] Master 127.0.198.62:46859 was elected leader, sending a full tablet report...
I20260929 18:36:06.497394 5906 ts_tablet_manager.cc:1428] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:36:06.498175 5758 heartbeater.cc:503] Master 127.0.198.62:46859 was elected leader, sending a full tablet report...
I20260929 18:36:06.499214 5908 raft_consensus.cc:348] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.499527 5908 raft_consensus.cc:374] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:06.499583 5908 raft_consensus.cc:729] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d667db46004a76aaa5df0ed53eca1c, State: Initialized, Role: FOLLOWER
I20260929 18:36:06.500092 5908 consensus_queue.cc:260] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.500751 5889 heartbeater.cc:503] Master 127.0.198.62:46859 was elected leader, sending a full tablet report...
I20260929 18:36:06.501092 5908 ts_tablet_manager.cc:1428] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:36:06.504274 5912 raft_consensus.cc:482] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:06.504433 5912 raft_consensus.cc:504] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.505462 5912 leader_election.cc:288] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d667db46004a76aaa5df0ed53eca1c (127.0.198.3:32799), eb02c1133e2c496bbe5df815cc3c2acd (127.0.198.1:38213)
I20260929 18:36:06.507225 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.506033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:48225 (local address 127.0.198.3:32799)
0929 18:36:06.506123 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.506128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.506140 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:06.506172 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.506174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.506202 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.506282 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.506880 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.506883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.506885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.506896 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.506918 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.506928 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.507013 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.507066 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.507104 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 18:36:06.507230 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.505671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:32799 (local address 127.0.198.2:48225)
0929 18:36:06.505932 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.505947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.506056 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.506307 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.506310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.506323 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.506735 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.506737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.506746 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.506775 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.507003 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.507006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.507008 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.507069 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.507104 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260929 18:36:06.507285 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.505679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38213 (local address 127.0.198.2:45177)
0929 18:36:06.506141 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.506152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.506224 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.506418 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.506420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.506439 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:06.506829 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:06.506831 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:06.506839 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:06.506858 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:06.507084 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:06.507087 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:06.507089 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:36:06.507151 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:36:06.507176 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":24,"thread_start_us":114,"threads_started":1}
I20260929 18:36:06.507282 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.505791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:45177 (local address 127.0.198.1:38213)
0929 18:36:06.505927 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.505930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.506224 (+ 294us) server_negotiation.cc:407] Connection header received
0929 18:36:06.506294 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.506297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.506330 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.506399 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.506960 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.506963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.506965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.506977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.507004 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.507014 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.507091 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.507150 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.507181 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:36:06.507874 5843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ca7af7180541208d0bddc076cce077" candidate_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d667db46004a76aaa5df0ed53eca1c" is_pre_election: true
I20260929 18:36:06.507980 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ca7af7180541208d0bddc076cce077" candidate_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" is_pre_election: true
I20260929 18:36:06.508174 5843 raft_consensus.cc:2454] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ec5f28bbb54f1e9ecae9e207ca6767 in term 0.
I20260929 18:36:06.508188 5581 raft_consensus.cc:2454] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ec5f28bbb54f1e9ecae9e207ca6767 in term 0.
I20260929 18:36:06.508590 5648 leader_election.cc:302] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ec5f28bbb54f1e9ecae9e207ca6767, eb02c1133e2c496bbe5df815cc3c2acd; no voters:
I20260929 18:36:06.508870 5912 raft_consensus.cc:2781] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:06.508958 5912 raft_consensus.cc:482] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:06.509024 5912 raft_consensus.cc:3037] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:06.510291 5912 raft_consensus.cc:504] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.510622 5912 leader_election.cc:288] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [CANDIDATE]: Term 1 election: Requested vote from peers 26d667db46004a76aaa5df0ed53eca1c (127.0.198.3:32799), eb02c1133e2c496bbe5df815cc3c2acd (127.0.198.1:38213)
I20260929 18:36:06.510962 5843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ca7af7180541208d0bddc076cce077" candidate_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d667db46004a76aaa5df0ed53eca1c"
I20260929 18:36:06.510993 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ca7af7180541208d0bddc076cce077" candidate_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02c1133e2c496bbe5df815cc3c2acd"
I20260929 18:36:06.511083 5843 raft_consensus.cc:3037] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:06.511093 5581 raft_consensus.cc:3037] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:36:06.511406 5759 tablet.cc:2369] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:06.512147 5843 raft_consensus.cc:2454] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ec5f28bbb54f1e9ecae9e207ca6767 in term 1.
I20260929 18:36:06.512243 5581 raft_consensus.cc:2454] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ec5f28bbb54f1e9ecae9e207ca6767 in term 1.
I20260929 18:36:06.512446 5646 leader_election.cc:302] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d667db46004a76aaa5df0ed53eca1c, 60ec5f28bbb54f1e9ecae9e207ca6767; no voters:
I20260929 18:36:06.512599 5912 raft_consensus.cc:2781] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:06.512825 5912 raft_consensus.cc:686] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 1 LEADER]: Becoming Leader. State: Replica: 60ec5f28bbb54f1e9ecae9e207ca6767, State: Running, Role: LEADER
I20260929 18:36:06.513191 5912 consensus_queue.cc:237] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } }
I20260929 18:36:06.516299 5439 catalog_manager.cc:5495] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ec5f28bbb54f1e9ecae9e207ca6767 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ec5f28bbb54f1e9ecae9e207ca6767" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36467 } health_report { overall_health: HEALTHY } } }
I20260929 18:36:06.526551 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.525075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.0.198.2:36467)
0929 18:36:06.525240 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.525266 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.525280 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:06.525356 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.525359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.525416 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.525482 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.526161 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.526165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.526168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.526180 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.526200 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.526208 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.526289 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.526348 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.526399 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:36:06.532429 5581 raft_consensus.cc:1261] T 14ca7af7180541208d0bddc076cce077 P eb02c1133e2c496bbe5df815cc3c2acd [term 1 FOLLOWER]: Refusing update from remote peer 60ec5f28bbb54f1e9ecae9e207ca6767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:06.532439 5843 raft_consensus.cc:1261] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 1 FOLLOWER]: Refusing update from remote peer 60ec5f28bbb54f1e9ecae9e207ca6767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:06.533159 5916 consensus_queue.cc:1035] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d667db46004a76aaa5df0ed53eca1c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:06.533349 5912 consensus_queue.cc:1035] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb02c1133e2c496bbe5df815cc3c2acd" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:06.549508 5437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:06.549722 5437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:06.552384 5437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc3cdcf8f7f848b78f4049274061e2e7: Sending DeleteTablet for 3 replicas of tablet 14ca7af7180541208d0bddc076cce077
I20260929 18:36:06.553315 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 14ca7af7180541208d0bddc076cce077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:06 UTC) from {username='slave'} at 127.0.0.1:34580
I20260929 18:36:06.553356 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 14ca7af7180541208d0bddc076cce077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:06 UTC) from {username='slave'} at 127.0.0.1:37474
I20260929 18:36:06.553593 5823 tablet_service.cc:1514] Processing DeleteTablet for tablet 14ca7af7180541208d0bddc076cce077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:06 UTC) from {username='slave'} at 127.0.0.1:47220
I20260929 18:36:06.553937 5929 tablet_replica.cc:330] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: stopping tablet replica
I20260929 18:36:06.554014 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 5499
I20260929 18:36:06.554839 5931 tablet_replica.cc:330] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: stopping tablet replica
I20260929 18:36:06.555181 5931 raft_consensus.cc:2229] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:06.555521 5931 raft_consensus.cc:2258] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:06.556054 5929 raft_consensus.cc:2229] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:06.556627 5929 raft_consensus.cc:2258] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:06.557212 5931 ts_tablet_manager.cc:1905] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:06.558287 5929 ts_tablet_manager.cc:1905] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:06.560168 5931 ts_tablet_manager.cc:1918] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:06.560369 5931 log.cc:1196] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/14ca7af7180541208d0bddc076cce077
I20260929 18:36:06.560748 5931 ts_tablet_manager.cc:1939] T 14ca7af7180541208d0bddc076cce077 P 26d667db46004a76aaa5df0ed53eca1c: Deleting consensus metadata
I20260929 18:36:06.560901 5929 ts_tablet_manager.cc:1918] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:06.561067 5929 log.cc:1196] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/14ca7af7180541208d0bddc076cce077
I20260929 18:36:06.561419 5929 ts_tablet_manager.cc:1939] T 14ca7af7180541208d0bddc076cce077 P 60ec5f28bbb54f1e9ecae9e207ca6767: Deleting consensus metadata
I20260929 18:36:06.562268 5424 catalog_manager.cc:4841] TS 26d667db46004a76aaa5df0ed53eca1c (127.0.198.3:32799): tablet 14ca7af7180541208d0bddc076cce077 (table test-table [id=d8239850740b47f0b832d04c8d376024]) successfully deleted
W20260929 18:36:06.562508 5426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:38213 (error 108)
I20260929 18:36:06.562574 5424 catalog_manager.cc:4841] TS 60ec5f28bbb54f1e9ecae9e207ca6767 (127.0.198.2:36467): tablet 14ca7af7180541208d0bddc076cce077 (table test-table [id=d8239850740b47f0b832d04c8d376024]) successfully deleted
I20260929 18:36:06.563328 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 5630
I20260929 18:36:06.563584 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.563280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38213 (local address 127.0.0.1:34584)
0929 18:36:06.563431 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.563500 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 18:36:06.566107 5426 catalog_manager.cc:4584] TS eb02c1133e2c496bbe5df815cc3c2acd (127.0.198.1:38213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14ca7af7180541208d0bddc076cce077: Network error: Client connection negotiation failed: client connection to 127.0.198.1:38213: connect: Connection refused (error 111)
I20260929 18:36:06.568377 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 5761
I20260929 18:36:06.573107 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 5408
2026-09-29T18:36:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (892 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 18:36:06.592885 792 mini_oidc.cc:152] Starting JWKS server
I20260929 18:36:06.594856 792 webserver.cc:460] Webserver started at https://127.0.0.1:38309/ using document root <none> and password file <none>
I20260929 18:36:06.595023 792 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:36:06.595252 792 webserver.cc:460] Webserver started at http://127.0.0.1:42237/ using document root <none> and password file <none>
2026-09-29T18:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:06Z Disabled control of system clock
I20260929 18:36:06.604198 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39485
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39485
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38309/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/testchainca.cert with env {}
W20260929 18:36:06.712603 5938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:06.712875 5938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:06.712913 5938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:06.716974 5938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:06.717047 5938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:06.717061 5938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:06.717077 5938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:06.717092 5938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:36:06.717105 5938 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:36:06.717118 5938 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38309/jwks.json
I20260929 18:36:06.722566 5938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38309/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:06.723584 5938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:06.724534 5938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:06.732928 5946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.732937 5944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.732935 5943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.732985 5938 server_base.cc:1034] running on GCE node
I20260929 18:36:06.733735 5938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:06.734876 5938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:06.736040 5938 hybrid_clock.cc:648] HybridClock initialized: now 1790706966736026 us; error 29 us; skew 500 ppm
I20260929 18:36:06.737906 5938 webserver.cc:460] Webserver started at http://127.0.198.62:36201/ using document root <none> and password file <none>
I20260929 18:36:06.738505 5938 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:06.738566 5938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:06.738749 5938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:06.740788 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "429851eb518e457ba7b4ad673b69bf9c"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "b610f00a9c70da6432507adcdcaab723"
server_key_iv: "771c67b962ecd10c652da0aa558b6353"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.741315 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "429851eb518e457ba7b4ad673b69bf9c"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "b610f00a9c70da6432507adcdcaab723"
server_key_iv: "771c67b962ecd10c652da0aa558b6353"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.744837 5938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:36:06.747130 5952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.748381 5938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:06.748479 5938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "429851eb518e457ba7b4ad673b69bf9c"
format_stamp: "Formatted at 2026-09-29 18:36:06 on dist-test-slave-q5qx"
server_key: "b610f00a9c70da6432507adcdcaab723"
server_key_iv: "771c67b962ecd10c652da0aa558b6353"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.748589 5938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:06.770512 5938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:06.771186 5938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:06.771366 5938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:06.778251 5938 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:39485
I20260929 18:36:06.778260 6004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:39485 every 8 connection(s)
I20260929 18:36:06.779177 5938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:06.779790 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 5938
I20260929 18:36:06.779911 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:06.780223 792 external_mini_cluster.cc:1428] Setting key 9c3ada20b65af04e187a50f6f6809d09
I20260929 18:36:06.782806 6005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:06.788137 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c: Bootstrap starting.
I20260929 18:36:06.789585 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.781655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.0.198.62:39485)
0929 18:36:06.782862 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.782882 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.782924 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:36:06.783571 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.783586 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.783976 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.784280 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:06.784710 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:06.785658 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:06.786562 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:06.787020 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:06.787193 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:06.787945 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:06.787950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:06.787963 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:06.788368 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:06.788423 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:06.788443 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:06.788521 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:06.788653 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 18:36:06.788894 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":692,"threads_started":1}
I20260929 18:36:06.790364 6005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:06.791065 6005 log.cc:824] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:06.792232 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:06.793341 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c: No bootstrap required, opened a new log
I20260929 18:36:06.796909 6005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429851eb518e457ba7b4ad673b69bf9c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39485 } }
I20260929 18:36:06.797130 6005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:06.797186 6005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429851eb518e457ba7b4ad673b69bf9c, State: Initialized, Role: FOLLOWER
I20260929 18:36:06.797577 6005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429851eb518e457ba7b4ad673b69bf9c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39485 } }
I20260929 18:36:06.797696 6005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:06.797756 6005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:06.797849 6005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:06.798730 6005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429851eb518e457ba7b4ad673b69bf9c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39485 } }
I20260929 18:36:06.799055 6005 leader_election.cc:302] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 429851eb518e457ba7b4ad673b69bf9c; no voters:
I20260929 18:36:06.799348 6005 leader_election.cc:288] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:06.799471 6010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:06.799665 6010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [term 1 LEADER]: Becoming Leader. State: Replica: 429851eb518e457ba7b4ad673b69bf9c, State: Running, Role: LEADER
I20260929 18:36:06.800001 6010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429851eb518e457ba7b4ad673b69bf9c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39485 } }
I20260929 18:36:06.800472 6005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:06.801626 6012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 429851eb518e457ba7b4ad673b69bf9c. Latest consensus state: current_term: 1 leader_uuid: "429851eb518e457ba7b4ad673b69bf9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429851eb518e457ba7b4ad673b69bf9c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39485 } } }
I20260929 18:36:06.801765 6012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:06.802176 6019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:06.802647 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "429851eb518e457ba7b4ad673b69bf9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429851eb518e457ba7b4ad673b69bf9c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39485 } } }
I20260929 18:36:06.802901 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:06.804877 6019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:06.809693 6019 catalog_manager.cc:1347] Generated new cluster ID: 993e97fc551249d39c02642c926c1480
I20260929 18:36:06.809764 6019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:06.837594 6019 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:06.838660 6019 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:06.846870 6019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 429851eb518e457ba7b4ad673b69bf9c: Generated new TSK 0
I20260929 18:36:06.847815 6019 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:36:06.880290 6035 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 18:36:06.880403 6035 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:36:06.880985 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.863703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.0.198.62:39485)
0929 18:36:06.863844 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:06.863849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:06.864074 (+ 225us) server_negotiation.cc:407] Connection header received
0929 18:36:06.864268 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:06.864271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:06.864330 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:06.864443 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:36:06.864721 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:06.865532 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:06.866451 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:06.866793 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:06.866907 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:06.867189 (+ 282us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:36:06.880571 (+ 13382us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:36:06.880847 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 18:36:06.881047 6008 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 18:36:06.882231 6034 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:06.863588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39485 (local address 127.0.0.1:35214)
0929 18:36:06.863990 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.864005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.864118 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.864483 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:06.864486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:06.864495 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:36:06.864616 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:06.864621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:06.865638 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:06.865642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:06.866339 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:06.866348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:06.866898 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:06.866901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:06.866962 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:06.867009 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:36:06.880830 (+ 13821us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:36:06.881040 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.198.62:39485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":100,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.198.62:39485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 18:36:06.882895 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 5938
2026-09-29T18:36:06Z chronyd exiting
I20260929 18:36:06.919214 792 test_util.cc:175] -----------------------------------------------
I20260929 18:36:06.919298 792 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706942746699-792-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 18:36:06.920331 792 mini_oidc.cc:152] Starting JWKS server
I20260929 18:36:06.921181 792 webserver.cc:460] Webserver started at https://127.0.0.1:41795/ using document root <none> and password file <none>
I20260929 18:36:06.921264 792 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:36:06.921479 792 webserver.cc:460] Webserver started at http://127.0.0.1:40757/ using document root <none> and password file <none>
2026-09-29T18:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:06Z Disabled control of system clock
I20260929 18:36:06.929335 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39289
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41795/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/testchainca.cert with env {}
W20260929 18:36:07.035912 6041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:07.036180 6041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:07.036217 6041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:07.040174 6041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:07.040230 6041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:07.040244 6041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:07.040259 6041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:07.040272 6041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:36:07.040287 6041 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:36:07.040314 6041 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41795/jwks.json
I20260929 18:36:07.045591 6041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41795/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:07.046609 6041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:07.047554 6041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:07.055975 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:07.055970 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:07.055987 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:07.056370 6041 server_base.cc:1034] running on GCE node
I20260929 18:36:07.056901 6041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:07.057941 6041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:07.059139 6041 hybrid_clock.cc:648] HybridClock initialized: now 1790706967059110 us; error 39 us; skew 500 ppm
I20260929 18:36:07.060895 6041 webserver.cc:460] Webserver started at http://127.0.198.62:34029/ using document root <none> and password file <none>
I20260929 18:36:07.061436 6041 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:07.061501 6041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:07.061679 6041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:07.063683 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "3043747374b4477f8bc155fad93ec140"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "6d91e9b7782c6345d2cd9b4dcee42956"
server_key_iv: "345ac7030c6bf2749933ca7922ff4859"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.064206 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "3043747374b4477f8bc155fad93ec140"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "6d91e9b7782c6345d2cd9b4dcee42956"
server_key_iv: "345ac7030c6bf2749933ca7922ff4859"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.067777 6041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:36:07.070112 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:07.071338 6041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:07.071445 6041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "3043747374b4477f8bc155fad93ec140"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "6d91e9b7782c6345d2cd9b4dcee42956"
server_key_iv: "345ac7030c6bf2749933ca7922ff4859"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.071545 6041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:07.089123 6041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:07.089841 6041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:07.090061 6041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:07.096973 6041 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:39289
I20260929 18:36:07.096992 6107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:39289 every 8 connection(s)
I20260929 18:36:07.097937 6041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:07.101143 6108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:07.104715 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6041
I20260929 18:36:07.104820 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:07.105053 792 external_mini_cluster.cc:1428] Setting key 47bbc39d5206496ff8e7b167e4ce037c
I20260929 18:36:07.107034 6108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140: Bootstrap starting.
I20260929 18:36:07.109828 6108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:07.110781 6108 log.cc:824] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:07.113154 6108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140: No bootstrap required, opened a new log
I20260929 18:36:07.115895 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.106439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.0.198.62:39289)
0929 18:36:07.107122 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:07.107145 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:07.107189 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:36:07.108035 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:07.108060 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:07.108608 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:07.108975 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:07.109453 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.110915 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.112023 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.112711 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.112867 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.113746 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:07.113752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:07.113769 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:07.114304 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:07.114374 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:07.114399 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:07.114502 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:07.114672 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:36:07.114946 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260929 18:36:07.117650 6108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3043747374b4477f8bc155fad93ec140" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39289 } }
I20260929 18:36:07.117935 6108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:07.118043 6108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3043747374b4477f8bc155fad93ec140, State: Initialized, Role: FOLLOWER
I20260929 18:36:07.118502 6108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3043747374b4477f8bc155fad93ec140" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39289 } }
I20260929 18:36:07.118670 6108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:07.118713 6108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:07.118875 6108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:07.119079 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:07.119953 6108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3043747374b4477f8bc155fad93ec140" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39289 } }
I20260929 18:36:07.120328 6108 leader_election.cc:302] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3043747374b4477f8bc155fad93ec140; no voters:
I20260929 18:36:07.120689 6108 leader_election.cc:288] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:07.120792 6113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:07.121003 6113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [term 1 LEADER]: Becoming Leader. State: Replica: 3043747374b4477f8bc155fad93ec140, State: Running, Role: LEADER
I20260929 18:36:07.121341 6113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3043747374b4477f8bc155fad93ec140" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39289 } }
I20260929 18:36:07.122033 6108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:07.122982 6115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3043747374b4477f8bc155fad93ec140. Latest consensus state: current_term: 1 leader_uuid: "3043747374b4477f8bc155fad93ec140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3043747374b4477f8bc155fad93ec140" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39289 } } }
I20260929 18:36:07.123135 6115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:07.123507 6122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:07.123462 6114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3043747374b4477f8bc155fad93ec140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3043747374b4477f8bc155fad93ec140" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39289 } } }
I20260929 18:36:07.124305 6114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:07.126758 6122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:07.132783 6122 catalog_manager.cc:1347] Generated new cluster ID: b9be1b563fb2435fa6d0ac48222a2984
I20260929 18:36:07.132869 6122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:07.151083 6122 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:07.152046 6122 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:07.157248 6122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3043747374b4477f8bc155fad93ec140: Generated new TSK 0
I20260929 18:36:07.157895 6122 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:36:07.182497 6138 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 18:36:07.182579 6138 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:36:07.183244 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.167325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.0.198.62:39289)
0929 18:36:07.167444 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:07.167449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:07.167629 (+ 180us) server_negotiation.cc:407] Connection header received
0929 18:36:07.167762 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:07.167765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:07.167821 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:07.167934 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:36:07.168100 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.168914 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.169742 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.170114 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.170222 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.170512 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:36:07.182830 (+ 12318us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:36:07.183088 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:36:07.183311 6111 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 18:36:07.183339 6137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:07.167216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39289 (local address 127.0.0.1:57242)
0929 18:36:07.167544 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.167556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:07.167658 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:07.167929 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:07.167931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:07.167938 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:36:07.168022 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:07.168026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.169018 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.169021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:07.169647 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:07.169655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.170257 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.170260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:07.170328 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.170351 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:36:07.183039 (+ 12688us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:36:07.183192 (+ 153us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.198.62:39289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260929 18:36:07.184137 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6041
2026-09-29T18:36:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 18:36:07.222050 792 mini_oidc.cc:152] Starting JWKS server
I20260929 18:36:07.222898 792 webserver.cc:460] Webserver started at https://127.0.0.1:44373/ using document root <none> and password file <none>
I20260929 18:36:07.223001 792 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:36:07.223300 792 webserver.cc:460] Webserver started at http://127.0.0.1:40061/ using document root <none> and password file <none>
2026-09-29T18:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:07Z Disabled control of system clock
I20260929 18:36:07.231829 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:40069
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:40069
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44373/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:36:07.338788 6144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:07.339061 6144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:07.339099 6144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:07.343084 6144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:07.343144 6144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:07.343158 6144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:07.343174 6144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:07.343189 6144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:36:07.343204 6144 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:36:07.343221 6144 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44373/jwks.json
I20260929 18:36:07.348017 6144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:40069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44373/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:07.349028 6144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:07.350385 6144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:07.358779 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:07.358760 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:07.358815 6144 server_base.cc:1034] running on GCE node
W20260929 18:36:07.358763 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:07.359655 6144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:07.360708 6144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:07.361884 6144 hybrid_clock.cc:648] HybridClock initialized: now 1790706967361869 us; error 27 us; skew 500 ppm
I20260929 18:36:07.363659 6144 webserver.cc:460] Webserver started at http://127.0.198.62:42207/ using document root <none> and password file <none>
I20260929 18:36:07.364238 6144 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:07.364310 6144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:07.364516 6144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:07.366623 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "9cb57d3420134d2ca5fa4088d61113a0"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "094a06a2d9ca0dd97c46974eccef8b6d"
server_key_iv: "00424194b94d1b69c5fbe4a1e710e7f8"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.367167 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "9cb57d3420134d2ca5fa4088d61113a0"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "094a06a2d9ca0dd97c46974eccef8b6d"
server_key_iv: "00424194b94d1b69c5fbe4a1e710e7f8"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.370769 6144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:07.373076 6158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:07.374301 6144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:07.374413 6144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "9cb57d3420134d2ca5fa4088d61113a0"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "094a06a2d9ca0dd97c46974eccef8b6d"
server_key_iv: "00424194b94d1b69c5fbe4a1e710e7f8"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.374508 6144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:07.394941 6144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:07.395818 6144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:07.395977 6144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:07.402403 6144 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:40069
I20260929 18:36:07.402412 6210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:40069 every 8 connection(s)
I20260929 18:36:07.403352 6144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:07.406517 6211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:07.407173 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6144
I20260929 18:36:07.407297 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:07.407536 792 external_mini_cluster.cc:1428] Setting key 23602c88f3e027f3566cbd64e6c5a147
I20260929 18:36:07.412513 6211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0: Bootstrap starting.
I20260929 18:36:07.415583 6211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:07.416512 6211 log.cc:824] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:07.418555 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.408833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.0.198.62:40069)
0929 18:36:07.409485 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:07.409500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:07.409555 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:36:07.410316 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:07.410336 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:07.410775 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:07.411093 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:07.411531 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.412895 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.414250 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.414935 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.416117 (+ 1182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.416378 (+ 261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:07.416384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:07.416403 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:07.417006 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:07.417069 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:07.417097 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:07.417188 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:07.417375 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 18:36:07.417637 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260929 18:36:07.421720 6211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0: No bootstrap required, opened a new log
I20260929 18:36:07.423090 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:07.425535 6211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb57d3420134d2ca5fa4088d61113a0" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40069 } }
I20260929 18:36:07.425755 6211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:07.425810 6211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb57d3420134d2ca5fa4088d61113a0, State: Initialized, Role: FOLLOWER
I20260929 18:36:07.426240 6211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb57d3420134d2ca5fa4088d61113a0" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40069 } }
I20260929 18:36:07.426366 6211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:07.426429 6211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:07.426527 6211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:07.427439 6211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb57d3420134d2ca5fa4088d61113a0" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40069 } }
I20260929 18:36:07.427769 6211 leader_election.cc:302] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cb57d3420134d2ca5fa4088d61113a0; no voters:
I20260929 18:36:07.428064 6211 leader_election.cc:288] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:07.428174 6216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:07.428401 6216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [term 1 LEADER]: Becoming Leader. State: Replica: 9cb57d3420134d2ca5fa4088d61113a0, State: Running, Role: LEADER
I20260929 18:36:07.428709 6216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb57d3420134d2ca5fa4088d61113a0" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40069 } }
I20260929 18:36:07.429277 6211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:07.430321 6218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cb57d3420134d2ca5fa4088d61113a0. Latest consensus state: current_term: 1 leader_uuid: "9cb57d3420134d2ca5fa4088d61113a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb57d3420134d2ca5fa4088d61113a0" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40069 } } }
I20260929 18:36:07.430387 6217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cb57d3420134d2ca5fa4088d61113a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb57d3420134d2ca5fa4088d61113a0" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40069 } } }
I20260929 18:36:07.430455 6218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:07.430482 6217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:07.430882 6225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:07.433944 6225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:07.439169 6225 catalog_manager.cc:1347] Generated new cluster ID: d6ab1f80ad6f4c9a91fa321b5495f1da
I20260929 18:36:07.439244 6225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:07.450093 6225 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:07.451022 6225 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:07.463719 6225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cb57d3420134d2ca5fa4088d61113a0: Generated new TSK 0
I20260929 18:36:07.464449 6225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:07.478006 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.472834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.0.198.62:40069)
0929 18:36:07.472963 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:07.472968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:07.473083 (+ 115us) server_negotiation.cc:407] Connection header received
0929 18:36:07.473239 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:07.473243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:07.473303 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:07.473421 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:36:07.473598 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.474521 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.475411 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.475757 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.475859 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.476132 (+ 273us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:36:07.477602 (+ 1470us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:36:07.477832 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:36:07.478080 6214 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 18:36:07.479750 6240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:07.472692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:40069 (local address 127.0.0.1:59268)
0929 18:36:07.473005 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.473017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:07.473110 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:07.473412 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:07.473414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:07.473424 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:36:07.473510 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:07.473515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.474654 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.474657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:07.475302 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:07.475311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.475877 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.475881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:07.475944 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.475967 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:36:07.477764 (+ 1797us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:36:07.479577 (+ 1813us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.198.62:40069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":1560,"thread_start_us":102,"threads_started":1}
I20260929 18:36:07.480199 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6144
E20260929 18:36:07.488644 6241 webserver.cc:533] Webserver: sslize failed:
E20260929 18:36:07.488720 6241 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T18:36:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T18:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:07Z Disabled control of system clock
I20260929 18:36:07.525781 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39145
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39145
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/testchainca.cert with env {}
W20260929 18:36:07.632395 6245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:07.632674 6245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:07.632714 6245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:07.637368 6245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:07.637458 6245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:07.637481 6245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:07.637503 6245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:07.637535 6245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:36:07.637564 6245 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:36:07.637590 6245 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 18:36:07.643342 6245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:07.644371 6245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:07.645335 6245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:07.653951 6245 server_base.cc:1034] running on GCE node
W20260929 18:36:07.653889 6250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:07.653904 6253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:07.653915 6251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:07.654659 6245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:07.655745 6245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:07.656929 6245 hybrid_clock.cc:648] HybridClock initialized: now 1790706967656904 us; error 41 us; skew 500 ppm
I20260929 18:36:07.658758 6245 webserver.cc:460] Webserver started at http://127.0.198.62:33873/ using document root <none> and password file <none>
I20260929 18:36:07.659307 6245 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:07.659365 6245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:07.659586 6245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:07.661549 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "7a1f9a8f30564861bef44163ee7225e9"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "227a66dce2f165ac3fea505d176c28dc"
server_key_iv: "f9857ed85e8e87189c04e9986d49147d"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.662097 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "7a1f9a8f30564861bef44163ee7225e9"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "227a66dce2f165ac3fea505d176c28dc"
server_key_iv: "f9857ed85e8e87189c04e9986d49147d"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.665593 6245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:07.668067 6259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:07.669481 6245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:07.669612 6245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "7a1f9a8f30564861bef44163ee7225e9"
format_stamp: "Formatted at 2026-09-29 18:36:07 on dist-test-slave-q5qx"
server_key: "227a66dce2f165ac3fea505d176c28dc"
server_key_iv: "f9857ed85e8e87189c04e9986d49147d"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.669711 6245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:07.683537 6245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:07.684212 6245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:07.684382 6245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:07.690973 6245 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:39145
I20260929 18:36:07.691061 6311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:39145 every 8 connection(s)
I20260929 18:36:07.691942 6245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:07.694924 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:07.700424 6312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9: Bootstrap starting.
I20260929 18:36:07.701340 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6245
I20260929 18:36:07.701467 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:07.701778 792 external_mini_cluster.cc:1428] Setting key 08504cf6c8db4f8615c07a773d4602f6
I20260929 18:36:07.703351 6312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:07.704062 6312 log.cc:824] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:07.706157 6312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9: No bootstrap required, opened a new log
I20260929 18:36:07.709750 6312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1f9a8f30564861bef44163ee7225e9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39145 } }
I20260929 18:36:07.710055 6312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:07.710122 6312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1f9a8f30564861bef44163ee7225e9, State: Initialized, Role: FOLLOWER
I20260929 18:36:07.710572 6312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1f9a8f30564861bef44163ee7225e9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39145 } }
I20260929 18:36:07.710695 6312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:07.710754 6312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:07.710846 6312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:07.710866 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.702864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.198.62:39145)
0929 18:36:07.703555 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:07.703570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:07.703607 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:36:07.704283 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:07.704302 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:07.704776 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:07.705136 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:07.705564 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.706593 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.707475 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.707996 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.708148 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.709000 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:07.709006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:07.709018 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:07.709412 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:07.709468 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:07.709488 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:07.709572 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:07.709709 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 18:36:07.709957 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":128,"threads_started":1}
I20260929 18:36:07.711989 6312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1f9a8f30564861bef44163ee7225e9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39145 } }
I20260929 18:36:07.712319 6312 leader_election.cc:302] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a1f9a8f30564861bef44163ee7225e9; no voters:
I20260929 18:36:07.712600 6312 leader_election.cc:288] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:07.712683 6317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:07.712847 6317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a1f9a8f30564861bef44163ee7225e9, State: Running, Role: LEADER
I20260929 18:36:07.713147 6317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1f9a8f30564861bef44163ee7225e9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39145 } }
I20260929 18:36:07.713828 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:07.713845 6312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:07.714715 6319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1f9a8f30564861bef44163ee7225e9. Latest consensus state: current_term: 1 leader_uuid: "7a1f9a8f30564861bef44163ee7225e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1f9a8f30564861bef44163ee7225e9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39145 } } }
I20260929 18:36:07.714867 6319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:07.715152 6318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a1f9a8f30564861bef44163ee7225e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1f9a8f30564861bef44163ee7225e9" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39145 } } }
I20260929 18:36:07.715237 6318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:07.718475 6332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:07.718547 6332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:07.718629 6326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:07.719331 6326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:07.724606 6326 catalog_manager.cc:1347] Generated new cluster ID: 3523402c2b924484b1457a57fd93df8a
I20260929 18:36:07.724673 6326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:07.751781 6326 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:07.752715 6326 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:07.763063 6326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a1f9a8f30564861bef44163ee7225e9: Generated new TSK 0
I20260929 18:36:07.763797 6326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:07.825196 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.776662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.0.198.62:39145)
0929 18:36:07.776770 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:07.776774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:07.776858 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:36:07.777017 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:07.777019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:07.777079 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:07.777167 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:36:07.777441 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.778041 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.779304 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.779530 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.779610 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.779915 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:36:07.824809 (+ 44894us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:36:07.825065 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49544: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 18:36:07.825280 6315 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49544: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 18:36:07.825501 6341 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:07.776447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39145 (local address 127.0.0.1:49544)
0929 18:36:07.776779 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.776794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:07.776909 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:07.777180 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:07.777183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:07.777191 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:36:07.777308 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:07.777315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.778185 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.778189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:07.779176 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:07.779186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:07.779645 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:07.779650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:07.779738 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:07.779765 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:36:07.825081 (+ 45316us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:36:07.825310 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.198.62:39145: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260929 18:36:07.826166 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6245
2026-09-29T18:36:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (326 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:08 dist-test-slave-q5qx krb5kdc[6353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:08 dist-test-slave-q5qx krb5kdc[6353](info): set up 2 sockets
Sep 29 18:36:08 dist-test-slave-q5qx krb5kdc[6353](info): commencing operation
krb5kdc: starting...
W20260929 18:36:09.022010 792 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:09 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:36:09.143031 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:34691
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:09.254480 6366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:09.254908 6366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:09.255000 6366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:09.261307 6366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:09.261374 6366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:09.261394 6366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:09.261411 6366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:09.261432 6366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:09.269337 6366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:09.270824 6366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:09.272253 6366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:09.284128 6372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:09.284319 6374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:09.284377 6371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:09.284518 6366 server_base.cc:1034] running on GCE node
I20260929 18:36:09.285226 6366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:09.286752 6366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:09.287954 6366 hybrid_clock.cc:648] HybridClock initialized: now 1790706969287934 us; error 39 us; skew 500 ppm
Sep 29 18:36:09 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:09.290959 6366 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:36:09.291883 6366 webserver.cc:460] Webserver started at http://127.0.198.62:34599/ using document root <none> and password file <none>
I20260929 18:36:09.292433 6366 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:09.292497 6366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:09.292687 6366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:09.294811 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "2b00fc4ff01241798c903c0acb3606fb"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "f92b494ee50fb3a933326cfe9c63c07b"
server_key_iv: "5128dc4d335fdbf5074ea2551e12486e"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.295356 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "2b00fc4ff01241798c903c0acb3606fb"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "f92b494ee50fb3a933326cfe9c63c07b"
server_key_iv: "5128dc4d335fdbf5074ea2551e12486e"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.299105 6366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:09.301498 6381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:09.302832 6366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:09.302950 6366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "2b00fc4ff01241798c903c0acb3606fb"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "f92b494ee50fb3a933326cfe9c63c07b"
server_key_iv: "5128dc4d335fdbf5074ea2551e12486e"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.303056 6366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:09.339773 6366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:09.342994 6366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:09.343197 6366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:09.350337 6366 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:34691
I20260929 18:36:09.350649 6433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:34691 every 8 connection(s)
I20260929 18:36:09.351354 6366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:09.354151 6434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:09.359709 6434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb: Bootstrap starting.
I20260929 18:36:09.359860 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6366
I20260929 18:36:09.359975 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:09.360239 792 external_mini_cluster.cc:1428] Setting key d3016364cf259983191846d4b649ea51
I20260929 18:36:09.362532 6434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:09.363301 6434 log.cc:824] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:09.365119 6434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb: No bootstrap required, opened a new log
I20260929 18:36:09.368705 6434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b00fc4ff01241798c903c0acb3606fb" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34691 } }
Sep 29 18:36:09 dist-test-slave-q5qx krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:09.368995 6434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:09.369053 6434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b00fc4ff01241798c903c0acb3606fb, State: Initialized, Role: FOLLOWER
I20260929 18:36:09.369475 6434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b00fc4ff01241798c903c0acb3606fb" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34691 } }
I20260929 18:36:09.369607 6434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:09.369671 6434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:09.369766 6434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:09.370808 6434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b00fc4ff01241798c903c0acb3606fb" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34691 } }
I20260929 18:36:09.371140 6434 leader_election.cc:302] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b00fc4ff01241798c903c0acb3606fb; no voters:
I20260929 18:36:09.371433 6434 leader_election.cc:288] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:09.371531 6439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:09.371688 6439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [term 1 LEADER]: Becoming Leader. State: Replica: 2b00fc4ff01241798c903c0acb3606fb, State: Running, Role: LEADER
I20260929 18:36:09.371996 6439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b00fc4ff01241798c903c0acb3606fb" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34691 } }
I20260929 18:36:09.372745 6434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:09.373245 6441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b00fc4ff01241798c903c0acb3606fb. Latest consensus state: current_term: 1 leader_uuid: "2b00fc4ff01241798c903c0acb3606fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b00fc4ff01241798c903c0acb3606fb" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34691 } } }
I20260929 18:36:09.373361 6441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:09.373615 6440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b00fc4ff01241798c903c0acb3606fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b00fc4ff01241798c903c0acb3606fb" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34691 } } }
I20260929 18:36:09.373710 6440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:09.375037 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.361630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.0.198.62:34691)
0929 18:36:09.362205 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:09.362221 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:09.362258 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:36:09.363006 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:09.363032 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:09.363410 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:09.363750 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:09.364542 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.365531 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.366385 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.366856 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.367010 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:09.369654 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:09.369661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:09.369680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:09.369711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:09.371409 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.371953 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.371960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.371966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.372109 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.372386 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.372390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.372392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.372822 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:09.373016 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:09.373640 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 18:36:09.373946 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":136,"threads_started":1}
W20260929 18:36:09.377630 6454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:09.377705 6454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:09.377775 6444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:09.378504 6444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:09.384155 6444 catalog_manager.cc:1347] Generated new cluster ID: a2e2364667834fac8d0ccc582c737368
I20260929 18:36:09.384234 6444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:09.409482 6444 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:09.410601 6444 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:09.419262 6444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b00fc4ff01241798c903c0acb3606fb: Generated new TSK 0
I20260929 18:36:09.420223 6444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:36:09.461998 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34691
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:09.575290 6460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:09.575589 6460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:09.575635 6460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:09.579900 6460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:09.579978 6460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:09.580039 6460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:09.586377 6460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:09.587579 6460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:09.589035 6460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:09.598737 6466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:09.598732 6465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:09.598732 6468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:09.599153 6460 server_base.cc:1034] running on GCE node
I20260929 18:36:09.599553 6460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:09.600203 6460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:09.601372 6460 hybrid_clock.cc:648] HybridClock initialized: now 1790706969601330 us; error 54 us; skew 500 ppm
Sep 29 18:36:09 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:09.604420 6460 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:36:09.605347 6460 webserver.cc:460] Webserver started at http://127.0.198.1:43985/ using document root <none> and password file <none>
I20260929 18:36:09.605906 6460 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:09.606005 6460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:09.606217 6460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:09.608461 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "36f8465de557472d94f9161800d40530"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "e89119127784db4eb22570fecfa70394"
server_key_iv: "ca12b31d2c618b10479eccfcf6265e26"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.609014 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "36f8465de557472d94f9161800d40530"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "e89119127784db4eb22570fecfa70394"
server_key_iv: "ca12b31d2c618b10479eccfcf6265e26"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.612783 6460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 18:36:09.615219 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:09.616411 6460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:09.616526 6460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "36f8465de557472d94f9161800d40530"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "e89119127784db4eb22570fecfa70394"
server_key_iv: "ca12b31d2c618b10479eccfcf6265e26"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.616667 6460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:09.639158 6460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:09.642900 6460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:09.643126 6460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:09.643852 6460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:09.644981 6460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:09.645032 6460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:09.645077 6460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:09.645104 6460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:09.654193 6460 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:33229
I20260929 18:36:09.654227 6588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:33229 every 8 connection(s)
I20260929 18:36:09.655115 6460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:09.659309 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6460
I20260929 18:36:09.659428 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:09.659683 792 external_mini_cluster.cc:1428] Setting key c2bb33385daef164980f5ad4e58d29be
Sep 29 18:36:09 dist-test-slave-q5qx krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:36:09.669175 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.656882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:58673 (local address 127.0.198.62:34691)
0929 18:36:09.657045 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:09.657050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:09.657937 (+ 887us) server_negotiation.cc:407] Connection header received
0929 18:36:09.659370 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:09.659374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:09.659429 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:09.659517 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:09.660774 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.661321 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.662477 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.662709 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.662796 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:09.665705 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:09.665708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:09.665711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:09.665736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:09.667065 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.667677 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.667680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.667681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.667755 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.668120 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.668122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.668123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.668295 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:09.668379 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:09.668933 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 18:36:09.669041 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:36:09.670080 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.657277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34691 (local address 127.0.198.1:58673)
0929 18:36:09.657779 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.657824 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:09.659134 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:09.659704 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:09.659713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:09.660092 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:09.660640 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:09.660654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.661439 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.661441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:09.662368 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:09.662377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.662867 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.662871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:09.663059 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:09.663689 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:09.663710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:09.665477 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:09.667195 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:09.667202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:09.667220 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:09.667578 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:09.667841 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:09.667845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:09.667846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:09.668027 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:09.668391 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:09.668398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:09.668725 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 18:36:09.668893 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 18:36:09.669164 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:36:09.671276 6589 heartbeater.cc:347] Connected to a master server at 127.0.198.62:34691
I20260929 18:36:09.671495 6589 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:09.671988 6589 heartbeater.cc:511] Master 127.0.198.62:34691 requested a full tablet report, sending...
I20260929 18:36:09.673450 6398 ts_manager.cc:194] Registered new tserver with Master: 36f8465de557472d94f9161800d40530 (127.0.198.1:33229)
I20260929 18:36:09.674625 6398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:58673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:36:09.690065 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34691
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:09.800483 6594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:09.800760 6594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:09.800803 6594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:09.805123 6594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:09.805209 6594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:09.805277 6594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:09.811477 6594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:09.812551 6594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:09.813567 6594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:09.823525 6602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:09.823534 6599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:09.823608 6594 server_base.cc:1034] running on GCE node
W20260929 18:36:09.823545 6600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:09.824198 6594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:09.824829 6594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:09.826010 6594 hybrid_clock.cc:648] HybridClock initialized: now 1790706969825973 us; error 46 us; skew 500 ppm
Sep 29 18:36:09 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:09.828684 6594 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:36:09.829559 6594 webserver.cc:460] Webserver started at http://127.0.198.2:35829/ using document root <none> and password file <none>
I20260929 18:36:09.830130 6594 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:09.830199 6594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:09.830400 6594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:09.832398 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "7c2f2ba2c51f499b8d0e48b4b9eae74e"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "8e6eaf7898f91b9a0c3c7672654deb9f"
server_key_iv: "c74b3f843a4d5cbace944cde1ea0a3c4"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.832966 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "7c2f2ba2c51f499b8d0e48b4b9eae74e"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "8e6eaf7898f91b9a0c3c7672654deb9f"
server_key_iv: "c74b3f843a4d5cbace944cde1ea0a3c4"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.836546 6594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:36:09.839033 6609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:09.840255 6594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:09.840349 6594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "7c2f2ba2c51f499b8d0e48b4b9eae74e"
format_stamp: "Formatted at 2026-09-29 18:36:09 on dist-test-slave-q5qx"
server_key: "8e6eaf7898f91b9a0c3c7672654deb9f"
server_key_iv: "c74b3f843a4d5cbace944cde1ea0a3c4"
server_key_version: "encryptionkey@0"
I20260929 18:36:09.840443 6594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:09.863054 6594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:09.866801 6594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:09.866990 6594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:09.867749 6594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:09.868623 6594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:09.868670 6594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:09.868712 6594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:09.868729 6594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:09.877956 6594 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:38927
I20260929 18:36:09.878011 6722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:38927 every 8 connection(s)
I20260929 18:36:09.878911 6594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:09.886523 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6594
I20260929 18:36:09.886642 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:09.886885 792 external_mini_cluster.cc:1428] Setting key a4448552b2d331b026165c584f67c1b5
Sep 29 18:36:09 dist-test-slave-q5qx krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:09.893110 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.880795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:43745 (local address 127.0.198.62:34691)
0929 18:36:09.880930 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:09.880934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:09.881905 (+ 971us) server_negotiation.cc:407] Connection header received
0929 18:36:09.883328 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:09.883331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:09.883396 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:09.883472 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:09.884737 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.885264 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.886635 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.886885 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.887002 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:09.889692 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:09.889695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:09.889698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:09.889727 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:09.891051 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.891550 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.891555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.891557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.891638 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.892009 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.892012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.892014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.892201 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:09.892286 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:09.892814 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 18:36:09.892941 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:36:09.893993 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.881250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34691 (local address 127.0.198.2:43745)
0929 18:36:09.881755 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.881801 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:09.883109 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:09.883639 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:09.883646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:09.884010 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:09.884522 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:09.884536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.885388 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.885391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:09.886481 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:09.886492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.887076 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.887081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:09.887288 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:09.887955 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:09.887972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:09.889522 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:09.891168 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:09.891174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:09.891193 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:09.891452 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:09.891759 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:09.891763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:09.891771 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:09.891919 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:09.892299 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:09.892306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:09.892578 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 18:36:09.892767 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:36:09.893121 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:36:09.895172 6723 heartbeater.cc:347] Connected to a master server at 127.0.198.62:34691
I20260929 18:36:09.895475 6723 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:09.895974 6723 heartbeater.cc:511] Master 127.0.198.62:34691 requested a full tablet report, sending...
I20260929 18:36:09.896996 6398 ts_manager.cc:194] Registered new tserver with Master: 7c2f2ba2c51f499b8d0e48b4b9eae74e (127.0.198.2:38927)
I20260929 18:36:09.897680 6398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:43745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:36:09.914440 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34691
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:10.023180 6728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:10.023453 6728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:10.023497 6728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:10.027927 6728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:10.028012 6728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:10.028086 6728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:10.033735 6728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:10.034832 6728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:10.035831 6728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:10.045822 6734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:10.045825 6736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:10.045863 6728 server_base.cc:1034] running on GCE node
W20260929 18:36:10.045825 6733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:10.046586 6728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:10.047241 6728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:10.048619 6728 hybrid_clock.cc:648] HybridClock initialized: now 1790706970048607 us; error 24 us; skew 500 ppm
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.051357 6728 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:36:10.052290 6728 webserver.cc:460] Webserver started at http://127.0.198.3:45177/ using document root <none> and password file <none>
I20260929 18:36:10.052820 6728 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:10.052881 6728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:10.053076 6728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:10.055135 6728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "7b7de8e1072e4246b64ddf51659bbbb2"
format_stamp: "Formatted at 2026-09-29 18:36:10 on dist-test-slave-q5qx"
server_key: "3357587a54406edc4d80dbbf9de60980"
server_key_iv: "58da6a38910b6633aefeba61606d2075"
server_key_version: "encryptionkey@0"
I20260929 18:36:10.055668 6728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "7b7de8e1072e4246b64ddf51659bbbb2"
format_stamp: "Formatted at 2026-09-29 18:36:10 on dist-test-slave-q5qx"
server_key: "3357587a54406edc4d80dbbf9de60980"
server_key_iv: "58da6a38910b6633aefeba61606d2075"
server_key_version: "encryptionkey@0"
I20260929 18:36:10.059334 6728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:10.061875 6743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:10.063123 6728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:10.063235 6728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "7b7de8e1072e4246b64ddf51659bbbb2"
format_stamp: "Formatted at 2026-09-29 18:36:10 on dist-test-slave-q5qx"
server_key: "3357587a54406edc4d80dbbf9de60980"
server_key_iv: "58da6a38910b6633aefeba61606d2075"
server_key_version: "encryptionkey@0"
I20260929 18:36:10.063330 6728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:10.074359 6728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:10.077149 6728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:10.077324 6728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:10.077876 6728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:10.078683 6728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:10.078737 6728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:10.078789 6728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:10.078814 6728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:10.087383 6728 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:40331
I20260929 18:36:10.087416 6856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:40331 every 8 connection(s)
I20260929 18:36:10.088243 6728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:10.090664 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6728
I20260929 18:36:10.090785 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:10.091097 792 external_mini_cluster.cc:1428] Setting key 197d72507e6a44f667aaf195b7cc23aa
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:10.102131 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.090094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:41585 (local address 127.0.198.62:34691)
0929 18:36:10.090223 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:10.090227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:10.091180 (+ 953us) server_negotiation.cc:407] Connection header received
0929 18:36:10.092736 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:10.092739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:10.092792 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:10.092865 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:10.094061 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.094611 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.095745 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.095952 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.096030 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:10.098543 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:10.098546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:10.098548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:10.098578 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:10.099978 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.100535 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.100538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.100540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.100619 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.100963 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.100965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.100967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.101138 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:10.101208 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:10.101813 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 18:36:10.101938 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:36:10.103087 6859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.090461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34691 (local address 127.0.198.3:41585)
0929 18:36:10.091026 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:10.091081 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:10.092521 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:10.093088 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:10.093095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:10.093442 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:10.093925 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:10.093939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.094710 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.094713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:10.095633 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:10.095644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.096093 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.096096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:10.096237 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:10.096825 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:10.096841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:10.098354 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:10.100109 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:10.100114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:10.100128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:10.100406 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:10.100719 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:10.100722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:10.100723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:10.100877 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:10.101241 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:10.101248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:10.101580 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 18:36:10.101738 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 18:36:10.102106 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":144,"threads_started":1}
I20260929 18:36:10.104466 6857 heartbeater.cc:347] Connected to a master server at 127.0.198.62:34691
I20260929 18:36:10.104727 6857 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:10.105286 6857 heartbeater.cc:511] Master 127.0.198.62:34691 requested a full tablet report, sending...
I20260929 18:36:10.106309 6398 ts_manager.cc:194] Registered new tserver with Master: 7b7de8e1072e4246b64ddf51659bbbb2 (127.0.198.3:40331)
I20260929 18:36:10.106865 6398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:41585
I20260929 18:36:10.117074 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/test-admin.keytab.
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.135731 792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 18:36:10.138624 6864 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:10.137138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.0.198.1:33229)
0929 18:36:10.137551 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:10.137566 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:10.137587 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:36:10.137634 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:10.137639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:10.137816 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:10.137998 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:10.138478 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45240: BlockingRecv error: recv got EOF from 127.0.0.1:45240 (error 108)
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":109,"threads_started":1}
W20260929 18:36:10.138813 792 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.139878 792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.140764 792 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:36:10.149890 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.141714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.0.198.1:33229)
0929 18:36:10.141835 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:10.141840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:10.141931 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:36:10.142070 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:10.142072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:10.142112 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:10.142171 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:10.142684 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.143471 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.144360 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.144725 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.144803 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:10.146961 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:10.146969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:10.146982 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:10.147018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:10.148404 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.148803 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.148808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.148809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.148879 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.149064 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.149066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.149067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.149282 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:10.149421 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:10.149583 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:36:10.149708 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 18:36:10.153685 6864 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:10.152663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.0.198.1:33229)
0929 18:36:10.152781 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:10.152785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:10.152880 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:36:10.153003 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:10.153006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:10.153045 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:10.153105 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:10.153578 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45254: BlockingRecv error: recv got EOF from 127.0.0.1:45254 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 18:36:10.153765 792 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.154700 792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.155455 792 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:36:10.163568 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.156197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.0.198.1:33229)
0929 18:36:10.156299 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:10.156303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:10.156388 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:36:10.156493 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:10.156496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:10.156539 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:10.156599 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:10.157078 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.157594 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.158384 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.158608 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.158692 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:10.160712 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:10.160715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:10.160718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:10.160739 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:10.161951 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.162447 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.162451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.162453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.162542 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.162790 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.162793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.162795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.162980 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:10.163089 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:10.163316 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 18:36:10.163420 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:36:10.166256 6864 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:10.165297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.0.198.1:33229)
0929 18:36:10.165417 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:10.165421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:10.165545 (+ 124us) server_negotiation.cc:407] Connection header received
0929 18:36:10.165649 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:10.165652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:10.165696 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:10.165756 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:10.166148 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45272: BlockingRecv error: recv got EOF from 127.0.0.1:45272 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:36:10.166385 792 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706942746699-792-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.167424 792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:10.168375 792 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:36:10.176663 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.169221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.0.198.1:33229)
0929 18:36:10.169364 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:10.169369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:10.169409 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:36:10.169515 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:10.169517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:10.169551 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:10.169612 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:10.170079 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.170600 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.171406 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:10.171632 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:10.171723 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:10.173847 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:10.173851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:10.173853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:10.173875 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:10.175233 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.175610 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.175613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.175614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.175693 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:10.175994 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:10.175996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:10.175997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:10.176144 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:10.176235 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:10.176430 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:36:10.176516 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:36:10.177613 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6460
I20260929 18:36:10.182757 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6594
I20260929 18:36:10.187201 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6728
I20260929 18:36:10.191565 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6366
2026-09-29T18:36:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2129 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6877](info): set up 2 sockets
Sep 29 18:36:10 dist-test-slave-q5qx krb5kdc[6877](info): commencing operation
krb5kdc: starting...
W20260929 18:36:11.146111 792 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:11 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab.
Entry for principal oryx/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab.
I20260929 18:36:11.260952 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35471
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab
--principal=oryx/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:11.369330 6890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:11.369585 6890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:11.369624 6890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:11.373548 6890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:11.373607 6890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:11.373625 6890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:11.373641 6890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:11.373653 6890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:11.378865 6890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab
--principal=oryx/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:11.379874 6890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:11.381160 6890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:11.389716 6895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:11.389863 6890 server_base.cc:1034] running on GCE node
W20260929 18:36:11.389721 6896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:11.389786 6898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:11.390713 6890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:11.392162 6890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:11.393334 6890 hybrid_clock.cc:648] HybridClock initialized: now 1790706971393296 us; error 56 us; skew 500 ppm
Sep 29 18:36:11 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:11.399921 6890 init.cc:377] Logged in from keytab as oryx/127.0.198.62@KRBTEST.COM (short username oryx)
I20260929 18:36:11.400983 6890 webserver.cc:460] Webserver started at http://127.0.198.62:43033/ using document root <none> and password file <none>
I20260929 18:36:11.401532 6890 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:11.401593 6890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:11.401760 6890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:11.403916 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "dc2c11a01b9d447a8c78d40f3d1e9726"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "fdf97a64225e0fb71dbc1262b95a8605"
server_key_iv: "736aaed6ad8e004f0c3be30b1d1b07e9"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.404448 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "dc2c11a01b9d447a8c78d40f3d1e9726"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "fdf97a64225e0fb71dbc1262b95a8605"
server_key_iv: "736aaed6ad8e004f0c3be30b1d1b07e9"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.408524 6890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:36:11.411209 6905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:11.412621 6890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:36:11.412768 6890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "dc2c11a01b9d447a8c78d40f3d1e9726"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "fdf97a64225e0fb71dbc1262b95a8605"
server_key_iv: "736aaed6ad8e004f0c3be30b1d1b07e9"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.412894 6890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:11.441623 6890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:11.445052 6890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:11.445231 6890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:11.452085 6890 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:35471
I20260929 18:36:11.452368 6957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:35471 every 8 connection(s)
I20260929 18:36:11.453104 6890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:11.455906 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:11.457333 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6890
I20260929 18:36:11.457484 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:11.457742 792 external_mini_cluster.cc:1428] Setting key d7d3504e0874259d379638489370ac2f
I20260929 18:36:11.462359 6958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: Bootstrap starting.
I20260929 18:36:11.465049 6958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:11.465900 6958 log.cc:824] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:36:11 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:11.467897 6958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: No bootstrap required, opened a new log
I20260929 18:36:11.471321 6958 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } }
I20260929 18:36:11.471572 6958 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:11.471647 6958 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2c11a01b9d447a8c78d40f3d1e9726, State: Initialized, Role: FOLLOWER
I20260929 18:36:11.472110 6958 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } }
I20260929 18:36:11.472257 6958 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:11.472321 6958 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:11.472421 6958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:11.472715 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.459156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.198.62:35471)
0929 18:36:11.459576 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:11.459595 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:11.459637 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:36:11.460454 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:11.460479 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:11.460937 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:11.461333 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:11.462318 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.463747 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.464727 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.465418 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.465570 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:11.468095 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:11.468102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:11.468116 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:11.468142 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:11.469746 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:11.470225 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:11.470230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:11.470235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:11.470358 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:11.470619 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:11.470622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:11.470623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:11.471026 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:11.471224 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:11.471499 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:36:11.471805 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
I20260929 18:36:11.473369 6958 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } }
I20260929 18:36:11.473717 6958 leader_election.cc:302] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2c11a01b9d447a8c78d40f3d1e9726; no voters:
I20260929 18:36:11.474056 6958 leader_election.cc:288] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:11.474134 6963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:11.474320 6963 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 1 LEADER]: Becoming Leader. State: Replica: dc2c11a01b9d447a8c78d40f3d1e9726, State: Running, Role: LEADER
I20260929 18:36:11.474665 6963 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } }
I20260929 18:36:11.475350 6958 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:11.475937 6963 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2c11a01b9d447a8c78d40f3d1e9726. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } }
I20260929 18:36:11.476171 6963 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:11.476369 6964 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } }
I20260929 18:36:11.476473 6964 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:11.476589 6971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:11.479643 6971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:11.485251 6971 catalog_manager.cc:1347] Generated new cluster ID: 001e9bcc2a3a42b1821a7b5a5f08184c
I20260929 18:36:11.485332 6971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:11.497219 6971 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:11.498332 6971 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:11.507216 6971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: Generated new TSK 0
I20260929 18:36:11.508018 6971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab.
Entry for principal oryx/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab.
I20260929 18:36:11.541208 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35471
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab
--principal=oryx/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:11.653781 6984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:11.654121 6984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:11.654197 6984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:11.660100 6984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:11.660176 6984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:11.660245 6984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:11.666167 6984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab
--principal=oryx/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:11.667305 6984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:11.668376 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:11.678764 6990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:11.678745 6989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:11.678751 6992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:11.679116 6984 server_base.cc:1034] running on GCE node
I20260929 18:36:11.679577 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:11.680241 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:11.681424 6984 hybrid_clock.cc:648] HybridClock initialized: now 1790706971681396 us; error 42 us; skew 500 ppm
Sep 29 18:36:11 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:11.684185 6984 init.cc:377] Logged in from keytab as oryx/127.0.198.1@KRBTEST.COM (short username oryx)
I20260929 18:36:11.685077 6984 webserver.cc:460] Webserver started at http://127.0.198.1:39457/ using document root <none> and password file <none>
I20260929 18:36:11.685612 6984 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:11.685671 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:11.685856 6984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:11.687989 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "f3bf34bc861148b6be55e008cbd1b809"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "58cb6540e9fddd1092e1c74c253df992"
server_key_iv: "49fbf4f7167c12584ff19464a2ce0235"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.688529 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "f3bf34bc861148b6be55e008cbd1b809"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "58cb6540e9fddd1092e1c74c253df992"
server_key_iv: "49fbf4f7167c12584ff19464a2ce0235"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.692444 6984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:36:11.695004 6999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:11.696345 6984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:11.696457 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "f3bf34bc861148b6be55e008cbd1b809"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "58cb6540e9fddd1092e1c74c253df992"
server_key_iv: "49fbf4f7167c12584ff19464a2ce0235"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.696576 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:11.709839 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:11.713187 6984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:11.713359 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:11.714003 6984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:11.714847 6984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:11.714895 6984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:11.714934 6984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:11.714962 6984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:11.723558 6984 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:37347
I20260929 18:36:11.723583 7112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:37347 every 8 connection(s)
I20260929 18:36:11.724458 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:11.727664 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 6984
I20260929 18:36:11.727784 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:11.728035 792 external_mini_cluster.cc:1428] Setting key 72e14f6ac3d7f73ab8cbed660f17d3b8
Sep 29 18:36:11 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:36:11.738231 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.726391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:37453 (local address 127.0.198.62:35471)
0929 18:36:11.726535 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:11.726539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:11.727435 (+ 896us) server_negotiation.cc:407] Connection header received
0929 18:36:11.728826 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:11.728830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:11.728885 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:11.728967 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:11.730203 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.730857 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.731983 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.732339 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.732443 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:11.734835 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:11.734838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:11.734840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:11.734871 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:11.736218 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:11.736736 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:11.736740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:11.736742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:11.736815 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:11.737116 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:11.737119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:11.737121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:11.737345 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:11.737441 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:11.737931 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 18:36:11.738075 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.2@KRBTEST.COM" created.
I20260929 18:36:11.739151 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.726733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.1:37453)
0929 18:36:11.727241 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:11.727287 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:11.728620 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:11.729152 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:11.729160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:11.729524 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:11.730071 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:11.730086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.730977 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.730981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:11.731868 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:11.731878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.732487 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.732491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:11.732637 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:11.733210 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:11.733227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:11.734672 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:11.736322 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:11.736328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:11.736346 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:11.736646 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:11.736893 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:11.736896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:11.736899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:11.737035 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:11.737464 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:11.737469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:11.737716 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 18:36:11.737890 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 18:36:11.738252 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260929 18:36:11.740437 7113 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35471
I20260929 18:36:11.740669 7113 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:11.741149 7113 heartbeater.cc:511] Master 127.0.198.62:35471 requested a full tablet report, sending...
I20260929 18:36:11.742733 6922 ts_manager.cc:194] Registered new tserver with Master: f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1:37347)
I20260929 18:36:11.743893 6922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:37453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab.
Entry for principal oryx/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab.
I20260929 18:36:11.754880 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35471
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab
--principal=oryx/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:11.871121 7118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:11.871416 7118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:11.871476 7118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:11.875600 7118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:11.875660 7118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:11.875746 7118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:11.881218 7118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab
--principal=oryx/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:11.882340 7118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:11.883425 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:11.892982 7123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:11.893020 7118 server_base.cc:1034] running on GCE node
W20260929 18:36:11.892982 7124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:11.892987 7126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:11.893620 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:11.894385 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:11.895552 7118 hybrid_clock.cc:648] HybridClock initialized: now 1790706971895536 us; error 31 us; skew 500 ppm
Sep 29 18:36:11 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:11.898285 7118 init.cc:377] Logged in from keytab as oryx/127.0.198.2@KRBTEST.COM (short username oryx)
I20260929 18:36:11.899206 7118 webserver.cc:460] Webserver started at http://127.0.198.2:37619/ using document root <none> and password file <none>
I20260929 18:36:11.899744 7118 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:11.899808 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:11.900008 7118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:11.902164 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "e72d1e9c100d4af8aca9ecd25f871875"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "01b8d975a9f6015630ab9c875478bb98"
server_key_iv: "fb7f49561caf7744df7bfc518227dfdb"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.902698 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "e72d1e9c100d4af8aca9ecd25f871875"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "01b8d975a9f6015630ab9c875478bb98"
server_key_iv: "fb7f49561caf7744df7bfc518227dfdb"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.906358 7118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:11.909018 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:11.910567 7118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:11.910681 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "e72d1e9c100d4af8aca9ecd25f871875"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "01b8d975a9f6015630ab9c875478bb98"
server_key_iv: "fb7f49561caf7744df7bfc518227dfdb"
server_key_version: "encryptionkey@0"
I20260929 18:36:11.910794 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:11.928062 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:11.931522 7118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:11.931751 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:11.932401 7118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:11.933255 7118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:11.933303 7118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:11.933357 7118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:11.933384 7118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:11.942173 7118 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:40181
I20260929 18:36:11.942265 7246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:40181 every 8 connection(s)
I20260929 18:36:11.943084 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:11.951556 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 7118
I20260929 18:36:11.951684 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:11.951978 792 external_mini_cluster.cc:1428] Setting key 2b92f35f83dc2b7c1a81b6ad7e5291b2
Sep 29 18:36:11 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:11.957491 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.944874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:53715 (local address 127.0.198.62:35471)
0929 18:36:11.945040 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:11.945044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:11.946025 (+ 981us) server_negotiation.cc:407] Connection header received
0929 18:36:11.947426 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:11.947430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:11.947490 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:11.947569 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:11.948867 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.949410 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.950664 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.950891 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.950980 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:11.953686 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:11.953690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:11.953693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:11.953727 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:11.955236 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:11.955866 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:11.955869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:11.955870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:11.955948 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:11.956310 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:11.956315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:11.956317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:11.956573 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:11.956668 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:11.957203 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 18:36:11.957328 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:36:11.958359 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.945253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.2:53715)
0929 18:36:11.945822 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:11.945878 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:11.947180 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:11.947773 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:11.947782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:11.948173 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:11.948704 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:11.948719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.949542 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.949545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:11.950537 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:11.950547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:11.951043 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:11.951047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:11.951222 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:11.951942 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:11.951959 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:11.953505 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:11.955362 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:11.955369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:11.955388 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:11.955757 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:11.956040 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:11.956043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:11.956044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:11.956194 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:11.956672 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:11.956677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:11.956956 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 18:36:11.957125 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:36:11.957432 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260929 18:36:11.959491 7247 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35471
I20260929 18:36:11.959770 7247 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:11.960353 7247 heartbeater.cc:511] Master 127.0.198.62:35471 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:36:11.961541 6922 ts_manager.cc:194] Registered new tserver with Master: e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181)
I20260929 18:36:11.962170 6922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:53715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab.
Entry for principal oryx/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab.
I20260929 18:36:11.979002 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35471
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab
--principal=oryx/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:12.090312 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:12.090560 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:12.090606 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:12.094666 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:12.094732 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:12.094794 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:12.100208 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab
--principal=oryx/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:12.101286 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:12.102345 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:12.111927 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.112038 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.112010 7252 server_base.cc:1034] running on GCE node
W20260929 18:36:12.111943 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.112596 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:12.113256 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:12.114478 7252 hybrid_clock.cc:648] HybridClock initialized: now 1790706972114452 us; error 47 us; skew 500 ppm
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:12.117612 7252 init.cc:377] Logged in from keytab as oryx/127.0.198.3@KRBTEST.COM (short username oryx)
I20260929 18:36:12.118584 7252 webserver.cc:460] Webserver started at http://127.0.198.3:37265/ using document root <none> and password file <none>
I20260929 18:36:12.119148 7252 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.119208 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.119405 7252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:12.121532 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "bcc940daaccb4500a084d85cf62e4d23"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "acf4d9b91ec7c7440edb283f20b0dbbc"
server_key_iv: "49eef37ec0bc37c838a423c891fa4758"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.122138 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "bcc940daaccb4500a084d85cf62e4d23"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "acf4d9b91ec7c7440edb283f20b0dbbc"
server_key_iv: "49eef37ec0bc37c838a423c891fa4758"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.125841 7252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:12.128563 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.129887 7252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:12.130046 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "bcc940daaccb4500a084d85cf62e4d23"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "acf4d9b91ec7c7440edb283f20b0dbbc"
server_key_iv: "49eef37ec0bc37c838a423c891fa4758"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.130157 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:12.142253 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:12.145364 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:12.145520 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:12.146100 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:12.146898 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:12.146942 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.146979 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:12.146999 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.155323 7380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:46357 every 8 connection(s)
I20260929 18:36:12.155357 7252 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:46357
I20260929 18:36:12.156243 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:12.165480 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 7252
I20260929 18:36:12.165602 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:12.165877 792 external_mini_cluster.cc:1428] Setting key 86def39334eded6e24f102150a9af196
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:12.170176 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.158018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:48571 (local address 127.0.198.62:35471)
0929 18:36:12.158163 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.158168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.159052 (+ 884us) server_negotiation.cc:407] Connection header received
0929 18:36:12.160406 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.160409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.160456 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.160530 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.161847 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.162433 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.163773 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.164006 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.164093 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.166891 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.166894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.166897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.166920 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.168192 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.168703 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.168707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.168709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.168807 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.169105 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.169108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.169110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.169287 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.169369 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.169858 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.170013 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:36:12.171002 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.158433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:48571)
0929 18:36:12.158891 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.158936 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.160125 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.160684 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.160692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.161062 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.161629 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.161644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.162598 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.162601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.163631 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.163641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.164265 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.164268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.164425 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.164987 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.165003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.166721 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.168342 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.168348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.168361 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.168605 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.168881 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.168884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.168886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.169021 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.169382 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.169389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.169637 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.169805 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.170175 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260929 18:36:12.172103 7381 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35471
I20260929 18:36:12.172362 7381 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:12.172830 7381 heartbeater.cc:511] Master 127.0.198.62:35471 requested a full tablet report, sending...
I20260929 18:36:12.173835 6922 ts_manager.cc:194] Registered new tserver with Master: bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3:46357)
I20260929 18:36:12.174502 6922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:48571
I20260929 18:36:12.179633 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.61.keytab.
Entry for principal oryx/127.0.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.61.keytab.
I20260929 18:36:12.205302 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:40923
--webserver_interface=127.0.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.61.keytab
--principal=oryx/127.0.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:35471,127.0.198.61:40923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:12.317571 7386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:12.317924 7386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:12.317996 7386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:12.322180 7386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:12.322250 7386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:12.322268 7386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:12.322284 7386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:12.322299 7386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:12.327837 7386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:35471,127.0.198.61:40923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.61.keytab
--principal=oryx/127.0.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:40923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
W20260929 18:36:12.328074 7386 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.198.62:35471,127.0.198.61:40923'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 18:36:12.328954 7386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:12.329931 7386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:12.338385 7391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.338469 7386 server_base.cc:1034] running on GCE node
W20260929 18:36:12.338497 7394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.338385 7392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.339193 7386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:12.340277 7386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:12.341436 7386 hybrid_clock.cc:648] HybridClock initialized: now 1790706972341410 us; error 44 us; skew 500 ppm
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:12.344135 7386 init.cc:377] Logged in from keytab as oryx/127.0.198.61@KRBTEST.COM (short username oryx)
I20260929 18:36:12.345058 7386 webserver.cc:460] Webserver started at http://127.0.198.61:39267/ using document root <none> and password file <none>
I20260929 18:36:12.345628 7386 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.345695 7386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.345897 7386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:12.347975 7386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/instance:
uuid: "f22c7915ba60461a9e05af9580de3851"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "ecfe38a864d6d3a4029e4b1e5ea0c8b2"
server_key_iv: "63fef9d8d38ad6556499fa6e63b2d949"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.348516 7386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal/instance:
uuid: "f22c7915ba60461a9e05af9580de3851"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "ecfe38a864d6d3a4029e4b1e5ea0c8b2"
server_key_iv: "63fef9d8d38ad6556499fa6e63b2d949"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.352196 7386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260929 18:36:12.354749 7401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.356083 7386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:12.356220 7386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
uuid: "f22c7915ba60461a9e05af9580de3851"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "ecfe38a864d6d3a4029e4b1e5ea0c8b2"
server_key_iv: "63fef9d8d38ad6556499fa6e63b2d949"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.356330 7386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:12.378568 7386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:12.381546 7386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:12.381719 7386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:12.388342 7386 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.61:40923
I20260929 18:36:12.388587 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.61:40923 every 8 connection(s)
I20260929 18:36:12.389294 7386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
I20260929 18:36:12.391439 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 7386
I20260929 18:36:12.392330 7454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:12.396226 7454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.198.61@KRBTEST.COM
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.61@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:12.405848 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.392916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.198.61:40923)
0929 18:36:12.393350 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.393365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.393400 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:36:12.394081 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.394098 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.394433 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.394730 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.395503 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.396465 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.397292 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.397818 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.398011 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.400881 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.400888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.400900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.400933 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.402804 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.403268 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.403275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.403282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.403423 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.403697 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.403713 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.403716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.404199 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.404412 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.404671 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.404982 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"spinlock_wait_cycles":15232,"thread_start_us":141,"threads_started":1}
I20260929 18:36:12.407117 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.398280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.0.198.62:35471)
0929 18:36:12.398416 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.398420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.398853 (+ 433us) server_negotiation.cc:407] Connection header received
0929 18:36:12.398976 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.398979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.399023 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.399088 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.399727 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.400265 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.401482 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.401702 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.401793 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.404044 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.404047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.404049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.404074 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.405429 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.405844 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.405847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.405848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.405921 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.406217 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.406221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.406222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.406444 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.406537 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.406838 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.406956 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:36:12.407142 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.397878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49210)
0929 18:36:12.398711 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.398765 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.398888 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.399147 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.399152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.399488 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.399609 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.399615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.400390 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.400394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.401370 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.401381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.401822 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.401826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.401965 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.402425 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.402436 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.403916 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.405533 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.405541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.405547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.405745 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.406045 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.406048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.406049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.406132 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.406539 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.406545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.406644 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.406752 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.406957 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":124,"threads_started":1}
I20260929 18:36:12.408531 7454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.61@KRBTEST.COM for oryx/127.0.198.61@KRBTEST.COM
I20260929 18:36:12.416334 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.408970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.0.1:47336)
0929 18:36:12.409088 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.409101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.409183 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.409366 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.409368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.409513 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.409590 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.409594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.410653 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.410656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.411157 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.411164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.411547 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.411550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.411668 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.412083 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.412093 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.413182 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.414876 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.414879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.414882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.415048 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.415275 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.415278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.415280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.415346 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.415676 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.415679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.415765 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.415846 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.416046 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":30}
I20260929 18:36:12.416334 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.409014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.0.198.61:40923)
0929 18:36:12.409111 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.409115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.409143 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:36:12.409252 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.409255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.409287 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.409349 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.409678 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.410205 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.411254 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.411463 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.411548 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.413289 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.413292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.413295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.413356 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.414762 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.415127 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.415130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.415132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.415203 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.415426 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.415429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.415430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.415595 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.415674 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.415895 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.416105 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":47}
I20260929 18:36:12.419397 7454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Bootstrap starting.
I20260929 18:36:12.421657 7454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:12.422492 7454 log.cc:824] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:12.424564 7454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: No bootstrap required, opened a new log
I20260929 18:36:12.427690 7454 raft_consensus.cc:348] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } }
I20260929 18:36:12.427912 7454 raft_consensus.cc:374] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:12.427979 7454 raft_consensus.cc:729] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22c7915ba60461a9e05af9580de3851, State: Initialized, Role: FOLLOWER
I20260929 18:36:12.428401 7454 consensus_queue.cc:260] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } }
I20260929 18:36:12.428872 7460 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } } }
I20260929 18:36:12.428988 7460 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:36:12.429551 7454 sys_catalog.cc:564] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:36:12.434389 7471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:12.434492 7471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:12.437040 7386 master_runner.cc:428] Detected that this master f22c7915ba60461a9e05af9580de3851 is joining an existing cluster
I20260929 18:36:12.437124 7386 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.198.61:40923
I20260929 18:36:12.448139 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.439642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.0.198.62:35471)
0929 18:36:12.439770 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.439774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.440202 (+ 428us) server_negotiation.cc:407] Connection header received
0929 18:36:12.440438 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.440441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.440492 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.440559 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.441011 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.441547 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.442521 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.442882 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.442988 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.444657 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.444661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.444663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.444691 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.446408 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.446848 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.446851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.446853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.446951 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.447281 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.447284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.447286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.447531 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.447617 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.447862 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.447983 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:36:12.448410 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.439674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.0.198.61:40923)
0929 18:36:12.439781 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.439785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.440505 (+ 720us) server_negotiation.cc:407] Connection header received
0929 18:36:12.440649 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.440652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.440687 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.440731 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.441310 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.442090 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.442934 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.443127 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.443204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.444315 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.444319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.444322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.444341 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.445657 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.446917 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.446920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.446922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.446989 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.447435 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.447438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.447440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.447658 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.447770 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.448024 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.448274 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":38}
I20260929 18:36:12.448705 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.439542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49218)
0929 18:36:12.440096 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.440110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.440222 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.440583 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.440585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.440787 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.440902 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.440908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.441642 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.441646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.442404 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.442413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.443131 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.443135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.443261 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.443874 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.443890 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.444503 (+ 613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.446512 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.446514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.446516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.446720 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.447038 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.447040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.447042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.447138 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.447621 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.447624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.447711 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.447815 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.448580 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":26,"thread_start_us":118,"threads_started":1}
I20260929 18:36:12.449018 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.439552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.0.1:47344)
0929 18:36:12.440421 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.440435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.440559 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.440866 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.440869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.441077 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.441196 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.441203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.442215 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.442218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.442846 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.442853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.443219 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.443222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.443302 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.443805 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.443821 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.444202 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.446539 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.446542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.446544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.446834 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.447075 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.447078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.447094 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.447298 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.447802 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.447805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.447881 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.448083 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.448876 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":915,"thread_start_us":121,"threads_started":1}
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.62@KRBTEST.COM for oryx/127.0.198.61@KRBTEST.COM
I20260929 18:36:12.464695 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.453739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.0.198.61:40923)
0929 18:36:12.453854 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.453859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.454383 (+ 524us) server_negotiation.cc:407] Connection header received
0929 18:36:12.454636 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.454639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.454696 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.454775 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.455529 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.456306 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.457548 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.457878 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.458019 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.461068 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.461072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.461075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.461101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.462714 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.463303 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.463306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.463308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.463392 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.463650 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.463653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.463655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.463912 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.464006 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.464412 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.464533 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:36:12.464763 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.453695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.0.1:47348)
0929 18:36:12.454260 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.454311 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.454483 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.454816 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.454822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.455281 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.455418 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.455426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.456409 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.456413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.457423 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.457434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.458035 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.458039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.458152 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.458775 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.458791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.460901 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.462832 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.462840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.462846 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.463186 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.463479 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.463482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.463484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.463565 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.464036 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.464043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.464195 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.464359 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.464575 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":115,"threads_started":1}
I20260929 18:36:12.466646 6922 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.198.61:40923: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f22c7915ba60461a9e05af9580de3851"
member_type: NON_VOTER
last_known_addr {
host: "127.0.198.61"
port: 40923
}
attrs {
promote: true
}
}
dest_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726"
cas_config_opid_index: -1
I20260929 18:36:12.467432 6922 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } }
I20260929 18:36:12.469321 7481 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f22c7915ba60461a9e05af9580de3851 (127.0.198.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } }
I20260929 18:36:12.470276 7428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:12.471395 7481 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.198.61:40923
I20260929 18:36:12.471674 7428 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Refusing update from remote peer dc2c11a01b9d447a8c78d40f3d1e9726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 18:36:12.471870 7481 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } }
I20260929 18:36:12.471916 7386 master.cc:537] Master@127.0.198.61:40923 shutting down...
I20260929 18:36:12.472406 7482 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.0.198.60@KRBTEST.COM; defaulting to no policy
I20260929 18:36:12.472064 7481 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:12.474584 6907 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.60@KRBTEST.COM" created.
W20260929 18:36:12.475919 6907 consensus_peers.cc:487] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 -> Peer f22c7915ba60461a9e05af9580de3851 (127.0.198.61:40923): Couldn't send request to peer f22c7915ba60461a9e05af9580de3851. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:36:12.476511 7386 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:12.476815 7386 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:12.476964 7386 tablet_replica.cc:330] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: stopping tablet replica
I20260929 18:36:12.482370 7386 master.cc:552] Master@127.0.198.61:40923 shutdown complete.
I20260929 18:36:12.482434 7386 master_runner.cc:305] Clearing existing system tablet
I20260929 18:36:12.482676 7386 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.482733 7386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.484081 7386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:12.484943 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.485200 7386 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:12.485323 7386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:12.485392 7386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
uuid: "f22c7915ba60461a9e05af9580de3851"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "ecfe38a864d6d3a4029e4b1e5ea0c8b2"
server_key_iv: "63fef9d8d38ad6556499fa6e63b2d949"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.487308 7386 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:12.490419 7386 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:36:12.490514 7386 log.cc:1196] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 18:36:12.491627 7386 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Deleting consensus metadata
I20260929 18:36:12.491801 7386 master_runner.cc:315] Copying system tablet from 127.0.198.62:35471
I20260929 18:36:12.492995 7386 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: tablet copy: Beginning tablet copy session from remote peer at address 127.0.198.62:35471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.60.keytab.
Entry for principal oryx/127.0.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.60.keytab.
I20260929 18:36:12.495487 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:40435
--webserver_interface=127.0.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.60.keytab
--principal=oryx/127.0.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:36:12.500555 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.493661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.0.198.62:35471)
0929 18:36:12.493789 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.493803 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.493904 (+ 101us) server_negotiation.cc:407] Connection header received
0929 18:36:12.494126 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.494128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.494176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.494244 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.494824 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.495369 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.496306 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.496524 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.496612 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.497652 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.497655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.497657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.497680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.499003 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.499382 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.499386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.499387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.499458 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.499691 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.499695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.499697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.499891 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.499994 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.500273 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.500410 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:36:12.500566 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.493533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49228)
0929 18:36:12.493801 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.493816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.493938 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.494281 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.494284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.494501 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.494641 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.494649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.495483 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.495487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.496185 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.496195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.496630 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.496634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.496726 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.497195 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.497209 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.497520 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.499104 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.499107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.499109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.499297 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.499535 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.499538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.499540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.499616 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.500035 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.500037 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.500118 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.500202 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.500361 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20260929 18:36:12.501052 6942 tablet_copy_service.cc:140] P dc2c11a01b9d447a8c78d40f3d1e9726: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f22c7915ba60461a9e05af9580de3851 ({username='oryx', principal='oryx/127.0.198.61@KRBTEST.COM'} at 127.0.0.1:49228)
I20260929 18:36:12.501212 6942 tablet_copy_service.cc:161] P dc2c11a01b9d447a8c78d40f3d1e9726: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f22c7915ba60461a9e05af9580de3851 at {username='oryx', principal='oryx/127.0.198.61@KRBTEST.COM'} at 127.0.0.1:49228: session id = f22c7915ba60461a9e05af9580de3851-00000000000000000000000000000000
I20260929 18:36:12.502692 6942 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:36:12.504106 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:36:12.505857 7386 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: tablet copy: Starting download of 0 data blocks...
I20260929 18:36:12.506058 7386 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: tablet copy: Starting download of 1 WAL segments...
I20260929 18:36:12.507544 7386 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:36:12.508998 6942 tablet_copy_service.cc:342] P dc2c11a01b9d447a8c78d40f3d1e9726: Request end of tablet copy session f22c7915ba60461a9e05af9580de3851-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.198.61@KRBTEST.COM'} at 127.0.0.1:49228
I20260929 18:36:12.509096 6942 tablet_copy_service.cc:433] P dc2c11a01b9d447a8c78d40f3d1e9726: ending tablet copy session f22c7915ba60461a9e05af9580de3851-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f22c7915ba60461a9e05af9580de3851
I20260929 18:36:12.515726 7386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:12.517549 7499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.517529 7496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.517917 7497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.518128 7386 server_base.cc:1034] running on GCE node
I20260929 18:36:12.518260 7386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:12.518522 7386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:12.519649 7386 hybrid_clock.cc:648] HybridClock initialized: now 1790706972519642 us; error 25 us; skew 500 ppm
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:12.520932 7386 init.cc:377] Logged in from keytab as oryx/127.0.198.61@KRBTEST.COM (short username oryx)
I20260929 18:36:12.521346 7386 webserver.cc:460] Webserver started at http://127.0.198.61:38683/ using document root <none> and password file <none>
I20260929 18:36:12.521555 7386 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.521605 7386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.522732 7386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:12.523553 7506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.523767 7386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:12.523839 7386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
uuid: "f22c7915ba60461a9e05af9580de3851"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "ecfe38a864d6d3a4029e4b1e5ea0c8b2"
server_key_iv: "63fef9d8d38ad6556499fa6e63b2d949"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.523919 7386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:12.541517 7386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:12.543270 7386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:12.547583 7386 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.61:40923
I20260929 18:36:12.547605 7558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.61:40923 every 8 connection(s)
I20260929 18:36:12.548251 7386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
I20260929 18:36:12.549885 7559 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:36:12.550726 7559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Bootstrap starting.
I20260929 18:36:12.558760 7559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:36:12.559041 7559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Bootstrap complete.
I20260929 18:36:12.559541 7559 raft_consensus.cc:348] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } }
I20260929 18:36:12.559655 7559 raft_consensus.cc:729] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f22c7915ba60461a9e05af9580de3851, State: Initialized, Role: LEARNER
I20260929 18:36:12.559755 7559 consensus_queue.cc:260] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } }
I20260929 18:36:12.560055 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } }
I20260929 18:36:12.560201 7559 sys_catalog.cc:564] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:12.560221 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:36:12.562722 7573 catalog_manager.cc:1259] Loaded cluster ID: 001e9bcc2a3a42b1821a7b5a5f08184c
I20260929 18:36:12.562837 7573 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: loading cluster ID for follower catalog manager: success
I20260929 18:36:12.564549 7573 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: acquiring CA information for follower catalog manager: success
I20260929 18:36:12.565646 7573 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:36:12.639324 7495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:12.639597 7495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:12.639657 7495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:12.643689 7495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:12.643756 7495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:12.643782 7495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:12.643803 7495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:12.643823 7495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:12.649339 7495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.60.keytab
--principal=oryx/127.0.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:40435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:12.650465 7495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:12.651477 7495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:12.660590 7578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.660681 7495 server_base.cc:1034] running on GCE node
W20260929 18:36:12.660616 7579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.660585 7581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.661526 7495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:12.662799 7495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:12.663995 7495 hybrid_clock.cc:648] HybridClock initialized: now 1790706972663921 us; error 89 us; skew 500 ppm
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:12.667150 7495 init.cc:377] Logged in from keytab as oryx/127.0.198.60@KRBTEST.COM (short username oryx)
I20260929 18:36:12.668200 7495 webserver.cc:460] Webserver started at http://127.0.198.60:33697/ using document root <none> and password file <none>
I20260929 18:36:12.668771 7495 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.668840 7495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.669028 7495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:12.671186 7495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/instance:
uuid: "4aa8d5aff8a54c7cbf92857373a82daf"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "5b059712896204dd390316d88ad67387"
server_key_iv: "9629531cf9daad21c791ecdf44e8a00c"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.671780 7495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal/instance:
uuid: "4aa8d5aff8a54c7cbf92857373a82daf"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "5b059712896204dd390316d88ad67387"
server_key_iv: "9629531cf9daad21c791ecdf44e8a00c"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.675782 7495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:36:12.678472 7588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.679829 7495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:12.679942 7495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
uuid: "4aa8d5aff8a54c7cbf92857373a82daf"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "5b059712896204dd390316d88ad67387"
server_key_iv: "9629531cf9daad21c791ecdf44e8a00c"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.680071 7495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:12.695374 7495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:12.698743 7495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:12.698940 7495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:12.706063 7495 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.60:40435
I20260929 18:36:12.706326 7640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.60:40435 every 8 connection(s)
I20260929 18:36:12.707049 7495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
I20260929 18:36:12.709996 7641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:12.713811 7641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:36:12.716739 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 7495
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.60@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.198.60@KRBTEST.COM
I20260929 18:36:12.728128 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.715877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.198.62:35471)
0929 18:36:12.716052 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.716056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.716867 (+ 811us) server_negotiation.cc:407] Connection header received
0929 18:36:12.718064 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.718067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.718120 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.718189 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.719407 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.720229 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.721566 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.721795 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.721874 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.724562 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.724566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.724569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.724601 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.726018 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.726570 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.726574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.726576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.726649 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.727030 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.727034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.727036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.727235 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.727301 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.727862 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.727973 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:36:12.728884 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.717671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.0.198.60:40435)
0929 18:36:12.718060 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.718074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.718112 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:36:12.718256 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.718261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.718465 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.718636 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.719412 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.720584 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.721629 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.722081 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.722172 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.724780 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.724786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.724798 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.724823 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.726615 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.727043 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.727049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.727051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.727144 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.727434 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.727437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.727438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.727716 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.728384 (+ 668us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.728481 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.728602 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":276,"spinlock_wait_cycles":8320,"thread_start_us":107,"threads_started":1}
I20260929 18:36:12.728888 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.716209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49232)
0929 18:36:12.716711 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.716767 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.717855 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.718299 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.718308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.718677 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.719236 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.719256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.720360 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.720362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.721424 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.721434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.721929 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.721933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.722114 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.722701 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.722717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.724395 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.726114 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.726123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.726132 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.726481 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.726750 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.726752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.726754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.726920 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.727379 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.727385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.727638 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.727810 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.728103 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":35,"spinlock_wait_cycles":2304,"thread_start_us":136,"threads_started":1}
I20260929 18:36:12.730752 7641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.60@KRBTEST.COM for oryx/127.0.198.61@KRBTEST.COM
I20260929 18:36:12.738667 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.731254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.0.1:47352)
0929 18:36:12.731389 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.731403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.731481 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.731906 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.731908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.732040 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.732184 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.732189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.733163 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.733166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.733742 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.733756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.734226 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.734229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.734307 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.734703 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.734713 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.735911 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.737342 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.737345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.737347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.737520 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.737746 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.737749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.737751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.737840 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.738206 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.738209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.738297 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.738384 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.738507 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:36:12.738700 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.731380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.0.198.61:40923)
0929 18:36:12.731647 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.731650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.731662 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:12.731700 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.731702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.731741 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.731813 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.732318 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.733018 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.733850 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.734128 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.734200 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.736038 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.736041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.736043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.736066 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.737242 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.737600 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.737604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.737605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.737671 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.737917 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.737920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.737922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.738126 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.738207 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.738443 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.738553 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":123,"threads_started":1}
I20260929 18:36:12.739367 7641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.60@KRBTEST.COM for oryx/127.0.198.60@KRBTEST.COM
I20260929 18:36:12.746112 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.739774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.0.1:33872)
0929 18:36:12.739860 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.739872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.739939 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.740115 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.740118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.740253 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.740319 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.740323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.740994 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.740998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.741488 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.741493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.741891 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.741895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.741960 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.742417 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.742427 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.743573 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.744867 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.744870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.744872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.745032 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.745240 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.745243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.745245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.745317 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.745620 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.745623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.745697 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.745784 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.745908 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 18:36:12.746131 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.739829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.0.198.60:40435)
0929 18:36:12.739920 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.739924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.739935 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:36:12.740003 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.740006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.740040 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.740099 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.740408 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.740903 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.741576 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.741806 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.741886 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.743685 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.743688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.743690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.743709 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.744780 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.745106 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.745109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.745110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.745170 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.745391 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.745394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.745395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.745546 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.745621 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.745827 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.745961 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":38}
I20260929 18:36:12.746402 7113 heartbeater.cc:503] Master 127.0.198.62:35471 was elected leader, sending a full tablet report...
I20260929 18:36:12.749851 7641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Bootstrap starting.
I20260929 18:36:12.752223 7641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:12.752925 7641 log.cc:824] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:12.755307 7641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: No bootstrap required, opened a new log
I20260929 18:36:12.758294 7641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } }
I20260929 18:36:12.758517 7641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:12.758555 7641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa8d5aff8a54c7cbf92857373a82daf, State: Initialized, Role: FOLLOWER
I20260929 18:36:12.758937 7641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } }
I20260929 18:36:12.759356 7647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } } }
I20260929 18:36:12.759475 7647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:36:12.760053 7641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:36:12.765003 7658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:12.765092 7658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:12.768249 7495 master_runner.cc:428] Detected that this master 4aa8d5aff8a54c7cbf92857373a82daf is joining an existing cluster
I20260929 18:36:12.768311 7495 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.198.60:40435
I20260929 18:36:12.778332 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.770863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.0.198.60:40435)
0929 18:36:12.770943 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.770947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.770958 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:36:12.770991 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.770994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.771031 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.771102 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.771582 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.772282 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.773345 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.773642 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.773729 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.775399 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.775403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.775405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.775426 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.776791 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.777193 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.777196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.777198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.777274 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.777490 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.777493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.777494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.777648 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.777736 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.778042 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.778161 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":25}
I20260929 18:36:12.778360 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.770420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.0.1:33882)
0929 18:36:12.770840 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.770852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.770932 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.771128 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.771130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.771353 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.771484 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.771491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.772375 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.772377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.773235 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.773243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.773861 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.773864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.773943 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.774898 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.774911 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.775293 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.776870 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.776872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.776874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.777106 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.777344 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.777347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.777349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.777420 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.777748 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.777751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.777847 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.777951 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.778111 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":29,"spinlock_wait_cycles":896,"thread_start_us":102,"threads_started":1}
I20260929 18:36:12.782106 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6984
I20260929 18:36:12.787976 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:37347
--local_ip_for_outbound_sockets=127.0.198.1
--tserver_master_addrs=127.0.198.62:35471
--webserver_port=39457
--webserver_interface=127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab
--principal=oryx/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:36:12.803825 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.770413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.0.1:47358)
0929 18:36:12.770687 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.770702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.770800 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.771589 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.771592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.771777 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.771886 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.771892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.772926 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.772929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.773847 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.773857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.774493 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.774497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.774612 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.798734 (+ 24122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.798751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.799299 (+ 548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.801900 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.801903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.801905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.802360 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.802696 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.802699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.802701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.802783 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.803218 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.803221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.803318 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.803432 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.803593 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":21,"spinlock_wait_cycles":3456,"thread_start_us":88,"threads_started":1}
I20260929 18:36:12.803829 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.771461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.0.198.62:35471)
0929 18:36:12.771592 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.771597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.772187 (+ 590us) server_negotiation.cc:407] Connection header received
0929 18:36:12.772349 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.772352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.772401 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.772450 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.773367 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.774100 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.775092 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.775434 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.775531 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.799496 (+ 23965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.799502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.799505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.799537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.801528 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.802506 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.802510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.802512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.802606 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.802909 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.802912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.802914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.803124 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.803207 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.803545 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.803669 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:36:12.804775 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.770755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.0.198.61:40923)
0929 18:36:12.770973 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.770977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.770988 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:36:12.771020 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.771023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.771064 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.771128 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.772007 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.772799 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.774020 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.774365 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.774496 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.799499 (+ 25003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.799504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.799507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.799538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.801675 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.802493 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.802497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.802499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.802587 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.802873 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.802876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.802878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.803097 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.803178 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.804479 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.804615 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 18:36:12.804379 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.771368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49236)
0929 18:36:12.772122 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.772133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.772239 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.772577 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.772579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.772768 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.773235 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.773243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.774218 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.774221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.774966 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.774976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.775657 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.775661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.775745 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.798629 (+ 22884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.798649 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.799253 (+ 604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.801892 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.801896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.801898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.802351 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.802729 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.802732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.802734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.802815 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.803273 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.803276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.803374 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.804048 (+ 674us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.804207 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":94,"threads_started":1}
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.62@KRBTEST.COM for oryx/127.0.198.60@KRBTEST.COM
I20260929 18:36:12.818435 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.809122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.0.198.60:40435)
0929 18:36:12.809228 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.809232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.809244 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:12.809375 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.809378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.809428 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.809497 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.809957 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.810745 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.811785 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.812166 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.812252 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.814756 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.814760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.814763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.814790 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.816599 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.817098 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.817102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.817104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.817197 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.817525 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.817528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.817530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.817762 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.817866 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.818168 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.818283 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:12.818436 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.809053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.0.1:33888)
0929 18:36:12.809172 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.809187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.809269 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.809536 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.809539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.809742 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.809852 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.809858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.810855 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.810858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.811674 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.811684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.812275 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.812279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.812371 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.812877 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.812899 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.814554 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.816723 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.816727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.816729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.816992 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.817329 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.817332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.817334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.817429 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.817845 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.817848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.817961 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.818105 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.818271 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:36:12.820546 6922 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.198.60:40435: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf"
member_type: NON_VOTER
last_known_addr {
host: "127.0.198.60"
port: 40435
}
attrs {
promote: true
}
}
dest_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726"
cas_config_opid_index: 5
I20260929 18:36:12.821266 6922 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } }
I20260929 18:36:12.822418 7482 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.823938 7482 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.198.60:40435
I20260929 18:36:12.824119 7482 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.824214 7482 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:12.825213 7495 master.cc:537] Master@127.0.198.60:40435 shutting down...
I20260929 18:36:12.825469 7615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:12.827298 7615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Refusing update from remote peer dc2c11a01b9d447a8c78d40f3d1e9726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:36:12.827952 7483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:12.830266 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.824603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.0.1:47366)
0929 18:36:12.825719 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.825735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.825826 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.826260 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.826264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.826281 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:12.826378 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.826384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.827434 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.827437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.828727 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.828738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.829813 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.829817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.829912 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.829965 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.830054 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.830133 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035}
W20260929 18:36:12.830518 6906 consensus_peers.cc:487] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 -> Peer 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60:40435): Couldn't send request to peer 4aa8d5aff8a54c7cbf92857373a82daf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:36:12.830679 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.824703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.0.198.61:40923)
0929 18:36:12.824832 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.824838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.825908 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 18:36:12.825962 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.825995 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.826057 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.826134 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:12.826488 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.827305 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.828869 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.829491 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.829601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.829752 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.830467 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.830547 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:12.831259 7533 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 LEARNER]: Refusing update from remote peer dc2c11a01b9d447a8c78d40f3d1e9726: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:36:12.831800 7483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 18:36:12.833928 7533 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.837023 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2c11a01b9d447a8c78d40f3d1e9726. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.837150 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:36:12.837242 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: NON_VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: true } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.837309 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:36:12.840186 7483 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: attempting to promote NON_VOTER f22c7915ba60461a9e05af9580de3851 to VOTER
I20260929 18:36:12.840705 7483 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } }
I20260929 18:36:12.841665 7495 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:12.841939 7495 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:12.842063 7495 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: stopping tablet replica
W20260929 18:36:12.843055 6906 consensus_peers.cc:487] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 -> Peer 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60:40435): Couldn't send request to peer 4aa8d5aff8a54c7cbf92857373a82daf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:36:12.843339 7533 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 LEARNER]: Refusing update from remote peer dc2c11a01b9d447a8c78d40f3d1e9726: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 18:36:12.843705 7482 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 18:36:12.845461 7482 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f22c7915ba60461a9e05af9580de3851 (127.0.198.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.847740 7533 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f22c7915ba60461a9e05af9580de3851 (127.0.198.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.848102 7482 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.848212 7482 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:12.848405 7483 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.848489 7483 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:12.849924 7495 master.cc:552] Master@127.0.198.60:40435 shutdown complete.
I20260929 18:36:12.849999 7495 master_runner.cc:305] Clearing existing system tablet
I20260929 18:36:12.850222 7495 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.850140 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.850278 7495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.850381 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:36:12.851459 7495 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:12.852376 7671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.852600 7495 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:12.852893 7495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:12.852972 7495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
uuid: "4aa8d5aff8a54c7cbf92857373a82daf"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "5b059712896204dd390316d88ad67387"
server_key_iv: "9629531cf9daad21c791ecdf44e8a00c"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.855365 7495 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:12.858040 7495 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:36:12.858224 7495 log.cc:1196] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 18:36:12.858543 7495 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Deleting consensus metadata
I20260929 18:36:12.858735 7495 master_runner.cc:315] Copying system tablet from 127.0.198.62:35471
I20260929 18:36:12.859972 7495 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.198.62:35471
I20260929 18:36:12.870466 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.860599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49238)
0929 18:36:12.860903 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.860918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:12.861102 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:12.861411 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:12.861415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:12.861631 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:12.861753 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.861761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.862865 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.862869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.863739 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:12.863750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.864339 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.864342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:12.864427 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.864975 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:12.864991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:12.865429 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:12.868654 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.868658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.868660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.868923 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.869287 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:12.869310 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:12.869312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:12.869407 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:12.869813 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:12.869816 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:12.869920 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 18:36:12.870060 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 18:36:12.870214 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260929 18:36:12.870473 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.860688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.0.198.62:35471)
0929 18:36:12.860815 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:12.860820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:12.860998 (+ 178us) server_negotiation.cc:407] Connection header received
0929 18:36:12.861216 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:12.861219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:12.861287 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:12.861389 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:12.861880 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.862755 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.863869 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:12.864193 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:12.864303 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:12.866823 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:12.866827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:12.866830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:12.866860 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:12.868538 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.869064 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.869069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.869071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.869162 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:12.869491 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:12.869494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:12.869496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:12.869717 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:12.869811 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:12.870123 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 18:36:12.870244 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:36:12.871237 6942 tablet_copy_service.cc:140] P dc2c11a01b9d447a8c78d40f3d1e9726: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4aa8d5aff8a54c7cbf92857373a82daf ({username='oryx', principal='oryx/127.0.198.60@KRBTEST.COM'} at 127.0.0.1:49238)
I20260929 18:36:12.871343 6942 tablet_copy_service.cc:161] P dc2c11a01b9d447a8c78d40f3d1e9726: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4aa8d5aff8a54c7cbf92857373a82daf at {username='oryx', principal='oryx/127.0.198.60@KRBTEST.COM'} at 127.0.0.1:49238: session id = 4aa8d5aff8a54c7cbf92857373a82daf-00000000000000000000000000000000
I20260929 18:36:12.872401 6942 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:36:12.873813 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:36:12.876493 7495 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: tablet copy: Starting download of 0 data blocks...
I20260929 18:36:12.876794 7495 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: tablet copy: Starting download of 1 WAL segments...
I20260929 18:36:12.878415 7495 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:36:12.880213 6942 tablet_copy_service.cc:342] P dc2c11a01b9d447a8c78d40f3d1e9726: Request end of tablet copy session 4aa8d5aff8a54c7cbf92857373a82daf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.198.60@KRBTEST.COM'} at 127.0.0.1:49238
I20260929 18:36:12.880306 6942 tablet_copy_service.cc:433] P dc2c11a01b9d447a8c78d40f3d1e9726: ending tablet copy session 4aa8d5aff8a54c7cbf92857373a82daf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4aa8d5aff8a54c7cbf92857373a82daf
I20260929 18:36:12.888056 7495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:12.889925 7678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.895138 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.895289 7681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.895555 7495 server_base.cc:1034] running on GCE node
I20260929 18:36:12.895718 7495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:12.896050 7495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:12.897189 7495 hybrid_clock.cc:648] HybridClock initialized: now 1790706972897171 us; error 36 us; skew 500 ppm
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:12.898921 7495 init.cc:377] Logged in from keytab as oryx/127.0.198.60@KRBTEST.COM (short username oryx)
I20260929 18:36:12.899413 7495 webserver.cc:460] Webserver started at http://127.0.198.60:45547/ using document root <none> and password file <none>
I20260929 18:36:12.899590 7495 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.899631 7495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.901124 7495 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:12.902146 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.902426 7495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:36:12.902506 7495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
uuid: "4aa8d5aff8a54c7cbf92857373a82daf"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "5b059712896204dd390316d88ad67387"
server_key_iv: "9629531cf9daad21c791ecdf44e8a00c"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.902582 7495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 18:36:12.932410 7667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:12.932682 7667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:12.932741 7667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:36:12.934762 7495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:12.936429 7495 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 18:36:12.937719 7667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:12.937788 7667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:12.937875 7667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:12.940781 7495 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.60:40435
I20260929 18:36:12.941537 7495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
I20260929 18:36:12.941550 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.60:40435 every 8 connection(s)
I20260929 18:36:12.943326 7744 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:36:12.944381 7744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Bootstrap starting.
I20260929 18:36:12.944504 7667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab
--principal=oryx/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:37347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=39457
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:12.945721 7667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:12.946995 7667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:12.955017 7744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:36:12.955304 7744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: Bootstrap complete.
I20260929 18:36:12.955811 7744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } }
I20260929 18:36:12.955936 7744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa8d5aff8a54c7cbf92857373a82daf, State: Initialized, Role: LEARNER
I20260929 18:36:12.956034 7744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } }
I20260929 18:36:12.956364 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:12.956475 7744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:12.956476 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: This master's current role is: LEARNER
W20260929 18:36:12.957929 7749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.957929 7751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:12.958065 7748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:12.958942 7667 server_base.cc:1034] running on GCE node
I20260929 18:36:12.959409 7667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:12.959745 7764 catalog_manager.cc:1259] Loaded cluster ID: 001e9bcc2a3a42b1821a7b5a5f08184c
I20260929 18:36:12.959796 7764 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: loading cluster ID for follower catalog manager: success
I20260929 18:36:12.960168 7667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:12.961366 7667 hybrid_clock.cc:648] HybridClock initialized: now 1790706972961343 us; error 32 us; skew 500 ppm
I20260929 18:36:12.961683 7764 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: acquiring CA information for follower catalog manager: success
I20260929 18:36:12.963063 7764 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 18:36:12 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:12.964511 7667 init.cc:377] Logged in from keytab as oryx/127.0.198.1@KRBTEST.COM (short username oryx)
I20260929 18:36:12.964517 7247 heartbeater.cc:503] Master 127.0.198.62:35471 was elected leader, sending a full tablet report...
I20260929 18:36:12.965574 7667 webserver.cc:460] Webserver started at http://127.0.198.1:39457/ using document root <none> and password file <none>
I20260929 18:36:12.966213 7667 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:12.966293 7667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:12.970392 7667 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:12.972647 7770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.973860 7667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:12.973999 7667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "f3bf34bc861148b6be55e008cbd1b809"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "58cb6540e9fddd1092e1c74c253df992"
server_key_iv: "49fbf4f7167c12584ff19464a2ce0235"
server_key_version: "encryptionkey@0"
I20260929 18:36:12.974541 7667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:12.986409 7667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:12.989645 7667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:12.989840 7667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:12.990473 7667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:12.991325 7667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:12.991372 7667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:12.991425 7667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:12.991454 7667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:13.000211 7667 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:37347
I20260929 18:36:13.000241 7883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:37347 every 8 connection(s)
I20260929 18:36:13.001216 7667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:13.008996 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 7667
I20260929 18:36:13.009197 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 7118
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.60@KRBTEST.COM
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.61@KRBTEST.COM
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:13.017037 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:40181
--local_ip_for_outbound_sockets=127.0.198.2
--tserver_master_addrs=127.0.198.62:35471
--webserver_port=37619
--webserver_interface=127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab
--principal=oryx/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:36:13.018226 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.003134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:42257 (local address 127.0.198.62:35471)
0929 18:36:13.003270 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.003274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.004597 (+ 1323us) server_negotiation.cc:407] Connection header received
0929 18:36:13.005190 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.005193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.005243 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.005302 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.006683 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.007512 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.011039 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.011377 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.011453 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.014367 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.014371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.014373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.014404 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.015997 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016536 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.016539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.016541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.016622 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016905 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.016908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.016910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.017115 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.017205 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.017924 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.018061 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:36:13.018226 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.003346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:40791 (local address 127.0.198.61:40923)
0929 18:36:13.003457 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.003461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.004217 (+ 756us) server_negotiation.cc:407] Connection header received
0929 18:36:13.005182 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.005185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.005244 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.005315 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.006682 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.007512 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.010462 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.010805 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.010879 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.013540 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.013543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.013545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.013571 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.015765 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016324 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.016328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.016330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.016414 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016848 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.016851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.016853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.017076 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.017146 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.017908 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.018061 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:36:13.018265 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.003009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:46659 (local address 127.0.198.60:40435)
0929 18:36:13.003411 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.003416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.004620 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 18:36:13.005188 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.005192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.005243 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.005302 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.007649 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.008329 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.009476 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.009819 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.009894 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.013941 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.013945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.013948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.014000 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.015595 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016319 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.016323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.016325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.016414 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016850 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.016854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.016856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.017076 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.017147 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.017915 (+ 768us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.018080 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":80,"threads_started":1}
I20260929 18:36:13.019879 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.003465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.198.1:40791)
0929 18:36:13.004000 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.004052 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.004981 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.005488 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.005495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.005931 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.006510 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.006526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.007687 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.007691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.010324 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.010334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.011020 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.011023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.011109 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.011547 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.011561 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.013346 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.015894 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.015898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.015900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.016215 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.016523 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.016527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.016749 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.017232 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.017238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.017632 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.018429 (+ 797us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.018848 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":286,"spinlock_wait_cycles":28416,"thread_start_us":122,"threads_started":1}
I20260929 18:36:13.020130 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.003465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.198.1:46659)
0929 18:36:13.004550 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.004561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.004989 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.005489 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.005498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.005992 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.006525 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.006531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.008438 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.008441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.009347 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.009358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.010110 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.010113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.010298 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.010893 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.010915 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.013779 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.015746 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.015752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.015772 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.016210 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.016522 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.016526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.016743 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.017230 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.017236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.017630 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.018430 (+ 800us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.018989 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":426,"spinlock_wait_cycles":116608,"thread_start_us":146,"threads_started":1}
I20260929 18:36:13.021143 7886 heartbeater.cc:347] Connected to a master server at 127.0.198.61:40923
I20260929 18:36:13.021160 7884 heartbeater.cc:347] Connected to a master server at 127.0.198.60:40435
I20260929 18:36:13.019879 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.003484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.1:42257)
0929 18:36:13.004485 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.004499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.004988 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.005489 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.005498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.005986 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.006522 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.006531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.007687 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.007691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.010911 (+ 3220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.010920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.011608 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.011611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.011708 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.012104 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.012117 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.014187 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.016137 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.016142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.016426 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.016736 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.016739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.016741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.016815 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.017339 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.017341 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.017632 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.018429 (+ 797us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.019121 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":584,"spinlock_wait_cycles":56320,"thread_start_us":143,"threads_started":1}
I20260929 18:36:13.021409 7886 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.021409 7884 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.022068 7884 heartbeater.cc:511] Master 127.0.198.60:40435 requested a full tablet report, sending...
I20260929 18:36:13.022068 7886 heartbeater.cc:511] Master 127.0.198.61:40923 requested a full tablet report, sending...
I20260929 18:36:13.022243 7885 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35471
I20260929 18:36:13.022310 7885 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.022477 7885 heartbeater.cc:511] Master 127.0.198.62:35471 requested a full tablet report, sending...
I20260929 18:36:13.023413 6922 ts_manager.cc:194] Re-registered known tserver with Master: f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1:37347)
I20260929 18:36:13.023509 7708 ts_manager.cc:194] Registered new tserver with Master: f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1:37347)
I20260929 18:36:13.023510 7523 ts_manager.cc:194] Registered new tserver with Master: f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1:37347)
I20260929 18:36:13.024163 6922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:42257
W20260929 18:36:13.133379 7892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:13.133657 7892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:13.133702 7892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:13.138023 7892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:13.138093 7892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:13.138159 7892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:13.143586 7892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab
--principal=oryx/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:40181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=37619
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:13.144639 7892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:13.145634 7892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:13.155676 7897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:13.155990 7892 server_base.cc:1034] running on GCE node
W20260929 18:36:13.159464 7900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:13.159780 7898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:13.160259 7892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:13.160967 7892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:13.162163 7892 hybrid_clock.cc:648] HybridClock initialized: now 1790706973162106 us; error 73 us; skew 500 ppm
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:13.165038 7892 init.cc:377] Logged in from keytab as oryx/127.0.198.2@KRBTEST.COM (short username oryx)
I20260929 18:36:13.166065 7892 webserver.cc:460] Webserver started at http://127.0.198.2:37619/ using document root <none> and password file <none>
I20260929 18:36:13.166694 7892 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:13.166774 7892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:13.170910 7892 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:13.173143 7907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:13.174535 7892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:13.174649 7892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "e72d1e9c100d4af8aca9ecd25f871875"
format_stamp: "Formatted at 2026-09-29 18:36:11 on dist-test-slave-q5qx"
server_key: "01b8d975a9f6015630ab9c875478bb98"
server_key_iv: "fb7f49561caf7744df7bfc518227dfdb"
server_key_version: "encryptionkey@0"
I20260929 18:36:13.175213 7892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:13.176492 7381 heartbeater.cc:503] Master 127.0.198.62:35471 was elected leader, sending a full tablet report...
I20260929 18:36:13.194660 7892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:13.197576 7892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:13.197760 7892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:13.198397 7892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:13.199271 7892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:13.199321 7892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:13.199373 7892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:13.199400 7892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:13.210120 7892 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:40181
I20260929 18:36:13.210147 8020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:40181 every 8 connection(s)
I20260929 18:36:13.211189 7892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:13.217483 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 7892
I20260929 18:36:13.217656 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 7252
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.61@KRBTEST.COM
I20260929 18:36:13.225018 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:46357
--local_ip_for_outbound_sockets=127.0.198.3
--tserver_master_addrs=127.0.198.62:35471
--webserver_port=37265
--webserver_interface=127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab
--principal=oryx/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.60@KRBTEST.COM
I20260929 18:36:13.227309 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.213024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:42255 (local address 127.0.198.62:35471)
0929 18:36:13.213152 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.213157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.214275 (+ 1118us) server_negotiation.cc:407] Connection header received
0929 18:36:13.215946 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.215950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.216003 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.216078 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.217116 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.217914 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.219811 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.220040 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.220127 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.222767 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.222771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.222774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.222803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.224779 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.225471 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.225474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.225477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.225557 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.226003 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.226006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.226008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.226218 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.226298 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.227008 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.227148 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:36:13.227309 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.213067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:35767 (local address 127.0.198.61:40923)
0929 18:36:13.213164 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.213168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.214909 (+ 1741us) server_negotiation.cc:407] Connection header received
0929 18:36:13.215438 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.215441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.215499 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.215566 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.217094 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.217908 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.219781 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.220118 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.220215 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.223090 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.223093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.223096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.223122 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.224704 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.225471 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.225474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.225476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.225557 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.226003 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.226006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.226008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.226218 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.226298 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.227008 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.227148 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:36:13.229010 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.213594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.2:42255)
0929 18:36:13.214122 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.214179 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.215230 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.216125 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.216128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.216334 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.216976 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.216983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.218101 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.218104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.219661 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.219672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.220178 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.220181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.220365 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.220913 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.220942 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.222551 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.224909 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.224912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.224914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.225360 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.225657 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.225660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.225662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.225874 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.226463 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.226465 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.226721 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.227503 (+ 782us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.227905 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":213,"spinlock_wait_cycles":28160,"thread_start_us":136,"threads_started":1}
I20260929 18:36:13.229363 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.214211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.198.2:37673)
0929 18:36:13.215051 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.215062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.215237 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.215851 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.215854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.216335 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.216935 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.216950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.222486 (+ 5536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.222489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.223332 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.223342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.224082 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.224085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.224171 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.224672 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.224686 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.226064 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.227981 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.227984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.227986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.228201 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.228456 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.228459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.228460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.228533 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.228873 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.228876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.228960 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.229042 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.229172 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":29,"spinlock_wait_cycles":84480,"thread_start_us":134,"threads_started":1}
I20260929 18:36:13.229364 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.214788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:37673 (local address 127.0.198.60:40435)
0929 18:36:13.214909 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.214913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.215117 (+ 204us) server_negotiation.cc:407] Connection header received
0929 18:36:13.215695 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.215697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.215739 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.215822 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.217068 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.217821 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.223466 (+ 5645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.223826 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.223908 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.226225 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.226229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.226232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.226264 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.227859 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.228286 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.228289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.228291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.228368 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.228614 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.228617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.228619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.228785 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.228867 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.229101 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.229226 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:36:13.229907 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.213594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.198.2:35767)
0929 18:36:13.214821 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.214833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.215239 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.215708 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.215715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.216342 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.216936 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.216952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.218101 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.218104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.219661 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.219672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.220339 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.220343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.220444 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.221031 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.221044 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.222938 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.224845 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.224851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.224869 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.225360 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.225657 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.225660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.225662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.225874 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.226363 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.226370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.226721 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.227503 (+ 782us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.229735 (+ 2232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"mutex_wait_us":261,"spinlock_wait_cycles":128384,"thread_start_us":696,"threads_started":1}
I20260929 18:36:13.230746 8021 heartbeater.cc:347] Connected to a master server at 127.0.198.60:40435
I20260929 18:36:13.231014 8021 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.231053 8023 heartbeater.cc:347] Connected to a master server at 127.0.198.61:40923
I20260929 18:36:13.231117 8023 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.230749 8022 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35471
I20260929 18:36:13.231506 8022 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.231606 8021 heartbeater.cc:511] Master 127.0.198.60:40435 requested a full tablet report, sending...
I20260929 18:36:13.231637 8022 heartbeater.cc:511] Master 127.0.198.62:35471 requested a full tablet report, sending...
I20260929 18:36:13.232184 8023 heartbeater.cc:511] Master 127.0.198.61:40923 requested a full tablet report, sending...
I20260929 18:36:13.232761 6922 ts_manager.cc:194] Re-registered known tserver with Master: e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181)
I20260929 18:36:13.232821 7523 ts_manager.cc:194] Registered new tserver with Master: e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181)
I20260929 18:36:13.232853 7708 ts_manager.cc:194] Registered new tserver with Master: e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181)
I20260929 18:36:13.233430 6922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:42255
W20260929 18:36:13.342949 8028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:13.343245 8028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:13.343309 8028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:13.347443 8028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:13.347515 8028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:13.347597 8028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:13.353154 8028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab
--principal=oryx/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:46357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=37265
--tserver_master_addrs=127.0.198.62:35471,127.0.198.61:40923,127.0.198.60:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:13.354317 8028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:13.355355 8028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:13.365491 8034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:13.365486 8033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:13.365489 8036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:13.365542 8028 server_base.cc:1034] running on GCE node
I20260929 18:36:13.366237 8028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:13.366963 8028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:13.368175 8028 hybrid_clock.cc:648] HybridClock initialized: now 1790706973368144 us; error 45 us; skew 500 ppm
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:13.371232 8028 init.cc:377] Logged in from keytab as oryx/127.0.198.3@KRBTEST.COM (short username oryx)
I20260929 18:36:13.372136 8028 webserver.cc:460] Webserver started at http://127.0.198.3:37265/ using document root <none> and password file <none>
I20260929 18:36:13.372663 8028 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:13.372731 8028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:13.376771 8028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:13.378958 8043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:13.380306 8028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:13.380407 8028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "bcc940daaccb4500a084d85cf62e4d23"
format_stamp: "Formatted at 2026-09-29 18:36:12 on dist-test-slave-q5qx"
server_key: "acf4d9b91ec7c7440edb283f20b0dbbc"
server_key_iv: "49eef37ec0bc37c838a423c891fa4758"
server_key_version: "encryptionkey@0"
I20260929 18:36:13.380934 8028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:13.396364 8028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:13.399468 8028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:13.399653 8028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:13.400218 8028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:13.401043 8028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:13.401096 8028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:13.401149 8028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:13.401175 8028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:13.409772 8028 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:46357
I20260929 18:36:13.409806 8156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:46357 every 8 connection(s)
I20260929 18:36:13.410861 8028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:13.414809 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 8028
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.60@KRBTEST.COM
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
Sep 29 18:36:13 dist-test-slave-q5qx krb5kdc[6877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.61@KRBTEST.COM
I20260929 18:36:13.431325 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.413337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:60153 (local address 127.0.198.61:40923)
0929 18:36:13.413451 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.413455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.414222 (+ 767us) server_negotiation.cc:407] Connection header received
0929 18:36:13.414904 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.414908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.414965 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.415017 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.416942 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.417764 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.419578 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.419921 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.419993 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.428034 (+ 8041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.428039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.428041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.428065 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.429715 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.430167 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.430170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.430172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.430250 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.430511 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.430514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.430516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.430714 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.430814 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.431077 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.431190 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":48}
I20260929 18:36:13.431387 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.413035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.198.3:43279)
0929 18:36:13.414070 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.414082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.414659 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.415431 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.415441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.416717 (+ 1276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.416827 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.416835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.418039 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.418042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.419446 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.419457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.420137 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.420150 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.420371 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.421204 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.421217 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.423532 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.427490 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.427497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.427517 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.428026 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.428315 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.428318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.428320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.428492 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.429317 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.429325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.429666 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.429864 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.430266 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":247,"spinlock_wait_cycles":42112,"thread_start_us":136,"threads_started":1}
I20260929 18:36:13.431952 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.416712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.0.198.62:35471)
0929 18:36:13.416969 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.416982 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.420282 (+ 3300us) server_negotiation.cc:407] Connection header received
0929 18:36:13.421548 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.421551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.421604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.423872 (+ 2268us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.423975 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.424722 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.425725 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.426099 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.426523 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.427614 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.427618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.427620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.427645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.429272 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.429847 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.429850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.429852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.429926 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.430242 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.430245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.430247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.430450 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.431132 (+ 682us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.431202 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.431322 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":67,"threads_started":1}
I20260929 18:36:13.431962 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.418148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.0.198.61:40923)
0929 18:36:13.419093 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.419098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.421357 (+ 2259us) server_negotiation.cc:407] Connection header received
0929 18:36:13.421399 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.421403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.421452 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.421534 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.422384 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.422966 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.424199 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.424613 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.424694 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.426477 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.426481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.426483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.426508 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.429666 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.430091 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.430095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.430096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.430167 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.430435 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.430438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.430440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.431019 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.431130 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.431723 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.431832 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":875,"thread_start_us":594,"threads_started":1}
I20260929 18:36:13.432166 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.413249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.198.3:60153)
0929 18:36:13.414128 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.414143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.414678 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.415431 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.415441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.416067 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.416764 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.416787 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.417944 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.417947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.419446 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.419457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.420148 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.420151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.420359 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.421082 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.421096 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.427892 (+ 6796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.429826 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.429829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.429831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.430065 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.430348 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.430351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.430353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.430426 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.430815 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.430817 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.430903 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.431508 (+ 605us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.431662 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":206,"spinlock_wait_cycles":11136,"thread_start_us":98,"threads_started":1}
I20260929 18:36:13.433027 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.412625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:43279 (local address 127.0.198.60:40435)
0929 18:36:13.412765 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.412770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.414175 (+ 1405us) server_negotiation.cc:407] Connection header received
0929 18:36:13.414909 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.414913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.414967 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.415017 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.417104 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.417853 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.419579 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.419922 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.420004 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.424036 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.424040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.424042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.424068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.427361 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.428125 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.428128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.428130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.428219 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.428919 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.428922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.428924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.429173 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.429263 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.432762 (+ 3499us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.432888 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:36:13.433152 8159 heartbeater.cc:347] Connected to a master server at 127.0.198.61:40923
I20260929 18:36:13.433398 8159 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.433998 8157 heartbeater.cc:347] Connected to a master server at 127.0.198.60:40435
I20260929 18:36:13.434078 8157 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.434391 8157 heartbeater.cc:511] Master 127.0.198.60:40435 requested a full tablet report, sending...
I20260929 18:36:13.434391 8159 heartbeater.cc:511] Master 127.0.198.61:40923 requested a full tablet report, sending...
I20260929 18:36:13.434762 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.413035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:44579)
0929 18:36:13.413686 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.413748 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.414660 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.415431 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.415441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.416056 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:13.416945 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.416953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.418263 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.418266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.419417 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.419428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.420137 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.420141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.420392 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.421002 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:13.421022 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:13.425648 (+ 4626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:13.427507 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.427510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.427517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.428626 (+ 1109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.431815 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:13.431819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:13.431821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:13.431906 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:13.432377 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:13.432380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:13.432464 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.433755 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.434165 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":406,"spinlock_wait_cycles":3328,"thread_start_us":129,"threads_started":1}
I20260929 18:36:13.435314 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.418061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.0.198.60:40435)
0929 18:36:13.419963 (+ 1902us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.419974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.419986 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:13.420020 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.420022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.420075 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.420146 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.421288 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.422080 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.423282 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.423612 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.423722 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.425308 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.425311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.425314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.425337 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.429282 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.433225 (+ 3943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.433229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.433231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.433323 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.433669 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.433673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.433675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.433896 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.434745 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.435025 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.435149 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1831,"thread_start_us":81,"threads_started":1}
I20260929 18:36:13.435461 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.412671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:44579 (local address 127.0.198.62:35471)
0929 18:36:13.412813 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.412818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.413846 (+ 1028us) server_negotiation.cc:407] Connection header received
0929 18:36:13.414910 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.414913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.414967 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.415017 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.417040 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.417773 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.419546 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.419871 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.419960 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.425801 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.425804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.425806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.425830 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.427417 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.428722 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.428726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.428728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.428811 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.432015 (+ 3204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.432018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.432020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.432245 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.432337 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.435202 (+ 2865us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.435326 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:36:13.435528 7522 ts_manager.cc:194] Registered new tserver with Master: bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3:46357)
I20260929 18:36:13.435527 7708 ts_manager.cc:194] Registered new tserver with Master: bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3:46357)
I20260929 18:36:13.436105 6921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:36:13.438179 8158 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35471
I20260929 18:36:13.438268 8158 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:13.438437 8158 heartbeater.cc:511] Master 127.0.198.62:35471 requested a full tablet report, sending...
I20260929 18:36:13.439020 6922 ts_manager.cc:194] Re-registered known tserver with Master: bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3:46357)
W20260929 18:36:13.439069 6921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:13.439620 6922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:44579
I20260929 18:36:13.454975 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.448409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.0.1:33916)
0929 18:36:13.448666 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.448681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.448800 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.449222 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.449225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.449242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:13.449342 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.449348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.450348 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.450351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.451441 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.451450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.454528 (+ 3078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.454531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.454663 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.454690 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.454748 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.454829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":19,"thread_start_us":121,"threads_started":1}
I20260929 18:36:13.455056 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.448757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.0.198.60:40435)
0929 18:36:13.448873 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.448877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.448892 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:36:13.448944 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.448947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.449008 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.449101 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:13.449474 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.450242 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.451625 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.454397 (+ 2772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.454664 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.454829 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.454866 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.454938 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:36:13.455971 7718 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260929 18:36:13.458498 8181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.451308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37347 (local address 127.0.0.1:49276)
0929 18:36:13.451561 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.451581 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.451684 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.453065 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.453067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.453085 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:13.453198 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.453204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.454520 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.454523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.456780 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.456791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.458077 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.458081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.458190 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.458214 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.458281 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.458356 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260929 18:36:13.458897 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2c11a01b9d447a8c78d40f3d1e9726. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:13.459040 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: This master's current role is: LEARNER
I20260929 18:36:13.459404 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.452068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.198.1:37347)
0929 18:36:13.452356 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.452369 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.452391 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:36:13.452472 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.452479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.452742 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.452918 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:13.453323 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.454409 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.456920 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.457743 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.457844 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.458015 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.458978 (+ 963us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.459268 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260929 18:36:13.462447 7818 tablet_service.cc:1467] Processing CreateTablet for tablet 81cfa9b6be92465ca74b965a395f2af3 (DEFAULT_TABLE table=test-table [id=7126c3d86b37492b9d47aa4abc6dc156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:13.463594 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfa9b6be92465ca74b965a395f2af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:13.467408 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.452480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:46357 (local address 127.0.0.1:50140)
0929 18:36:13.452865 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.452877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.452943 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.455297 (+ 2354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.455301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.455319 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:13.455399 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.455404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.464039 (+ 8635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.464042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.465516 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.465529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.466925 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.466928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.467041 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.467071 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.467175 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.467262 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":75,"threads_started":1}
I20260929 18:36:13.468001 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.452634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.198.3:46357)
0929 18:36:13.453849 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.453859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.453879 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:36:13.453950 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.453955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.454170 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.459340 (+ 5170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:13.459510 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.463958 (+ 4448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.465804 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.466761 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.466891 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.467062 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.467205 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.467304 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":86,"threads_started":1}
I20260929 18:36:13.468442 7483 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: NON_VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: true } } }
I20260929 18:36:13.468573 7483 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:13.469525 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.451754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:40181 (local address 127.0.0.1:33104)
0929 18:36:13.451979 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.451992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.452071 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.454320 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.454323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.454339 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:13.454436 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.454442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.463928 (+ 9486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.463932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.467758 (+ 3826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.467769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.469096 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.469099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.469200 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.469245 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.469315 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.469392 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260929 18:36:13.469946 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.451929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.0.198.2:40181)
0929 18:36:13.453228 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.453239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.453262 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:36:13.453339 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.453344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.453551 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.454585 (+ 1034us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:13.454747 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.463760 (+ 9013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.467915 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.468751 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.468860 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.469028 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.469703 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.469811 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194,"thread_start_us":95,"threads_started":1}
I20260929 18:36:13.470141 8175 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: attempting to promote NON_VOTER 4aa8d5aff8a54c7cbf92857373a82daf to VOTER
I20260929 18:36:13.470501 8175 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } }
I20260929 18:36:13.471828 7718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 LEARNER]: Refusing update from remote peer dc2c11a01b9d447a8c78d40f3d1e9726: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260929 18:36:13.472072 7533 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Refusing update from remote peer dc2c11a01b9d447a8c78d40f3d1e9726: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260929 18:36:13.472591 7483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260929 18:36:13.471143 8091 tablet_service.cc:1467] Processing CreateTablet for tablet 81cfa9b6be92465ca74b965a395f2af3 (DEFAULT_TABLE table=test-table [id=7126c3d86b37492b9d47aa4abc6dc156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:13.473287 8091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfa9b6be92465ca74b965a395f2af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:13.474264 7483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260929 18:36:13.474674 7483 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.474450 7952 tablet_service.cc:1467] Processing CreateTablet for tablet 81cfa9b6be92465ca74b965a395f2af3 (DEFAULT_TABLE table=test-table [id=7126c3d86b37492b9d47aa4abc6dc156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:13.475553 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfa9b6be92465ca74b965a395f2af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:13.476183 7483 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.476305 7483 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:13.476404 7483 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.476482 7483 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:13.477245 7533 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.479573 8191 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.479758 8191 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:36:13.481834 8192 tablet_bootstrap.cc:492] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Bootstrap starting.
I20260929 18:36:13.482985 8193 tablet_bootstrap.cc:492] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Bootstrap starting.
I20260929 18:36:13.483088 8180 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.483234 8180 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:13.483912 7718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.485426 8193 tablet_bootstrap.cc:654] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:13.485426 8192 tablet_bootstrap.cc:654] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:13.485805 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:13.485918 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:36:13.486114 8194 tablet_bootstrap.cc:492] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Bootstrap starting.
I20260929 18:36:13.486500 8193 log.cc:824] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:13.486588 8192 log.cc:824] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:13.488243 8194 tablet_bootstrap.cc:654] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:13.488260 8193 tablet_bootstrap.cc:492] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: No bootstrap required, opened a new log
I20260929 18:36:13.488440 8193 ts_tablet_manager.cc:1397] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:36:13.488502 8192 tablet_bootstrap.cc:492] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: No bootstrap required, opened a new log
I20260929 18:36:13.488692 8192 ts_tablet_manager.cc:1397] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:36:13.489033 8194 log.cc:824] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:13.491040 8194 tablet_bootstrap.cc:492] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: No bootstrap required, opened a new log
I20260929 18:36:13.491246 8194 ts_tablet_manager.cc:1397] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:36:13.492256 8193 raft_consensus.cc:348] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.492475 8193 raft_consensus.cc:374] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:13.492524 8193 raft_consensus.cc:729] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc940daaccb4500a084d85cf62e4d23, State: Initialized, Role: FOLLOWER
I20260929 18:36:13.492946 8193 consensus_queue.cc:260] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.493148 8192 raft_consensus.cc:348] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.493405 8192 raft_consensus.cc:374] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:13.493453 8192 raft_consensus.cc:729] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bf34bc861148b6be55e008cbd1b809, State: Initialized, Role: FOLLOWER
I20260929 18:36:13.493783 8158 heartbeater.cc:503] Master 127.0.198.62:35471 was elected leader, sending a full tablet report...
I20260929 18:36:13.493963 8193 ts_tablet_manager.cc:1428] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:36:13.494391 8192 consensus_queue.cc:260] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.495404 8192 ts_tablet_manager.cc:1428] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:36:13.497176 7885 heartbeater.cc:503] Master 127.0.198.62:35471 was elected leader, sending a full tablet report...
I20260929 18:36:13.498675 8194 raft_consensus.cc:348] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.498989 8194 raft_consensus.cc:374] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:13.499051 8194 raft_consensus.cc:729] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72d1e9c100d4af8aca9ecd25f871875, State: Initialized, Role: FOLLOWER
I20260929 18:36:13.499536 8194 consensus_queue.cc:260] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.500214 8022 heartbeater.cc:503] Master 127.0.198.62:35471 was elected leader, sending a full tablet report...
I20260929 18:36:13.500566 8194 ts_tablet_manager.cc:1428] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260929 18:36:13.503176 7887 tablet.cc:2369] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:13.612895 8199 raft_consensus.cc:482] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:13.613114 8199 raft_consensus.cc:504] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.614332 8199 leader_election.cc:288] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181), bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3:46357)
I20260929 18:36:13.618160 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.614644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:60673 (local address 127.0.198.2:40181)
0929 18:36:13.614770 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.614775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.614932 (+ 157us) server_negotiation.cc:407] Connection header received
0929 18:36:13.615069 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.615072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.615118 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.615195 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:13.615488 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.615997 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.617237 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.617630 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.617724 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.617789 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.617948 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.618040 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:13.618325 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.614672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:53417 (local address 127.0.198.3:46357)
0929 18:36:13.614875 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.614880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.614960 (+ 80us) server_negotiation.cc:407] Connection header received
0929 18:36:13.615094 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.615096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.615137 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.615193 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:13.615476 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.615993 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.617276 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.617774 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.617869 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.617927 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.618138 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.618205 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 18:36:13.618345 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.614589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:46357 (local address 127.0.198.1:53417)
0929 18:36:13.614867 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.614880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.614983 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.615215 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.615218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.615289 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:13.615380 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.615384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.616158 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.616160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.617104 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.617113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.617884 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.617887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.618007 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.618029 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.618117 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.618186 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":86,"spinlock_wait_cycles":768,"thread_start_us":70,"threads_started":1}
I20260929 18:36:13.618178 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.614577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:40181 (local address 127.0.198.1:60673)
0929 18:36:13.614837 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.614850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:13.614958 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:13.615210 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:13.615212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:13.615238 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:13.615352 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.615359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.616103 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.616106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.617107 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:13.617116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.617742 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.617745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:13.617838 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.617860 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:36:13.617926 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:36:13.618026 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":55,"spinlock_wait_cycles":1408,"thread_start_us":81,"threads_started":1}
I20260929 18:36:13.619127 8111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfa9b6be92465ca74b965a395f2af3" candidate_uuid: "f3bf34bc861148b6be55e008cbd1b809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc940daaccb4500a084d85cf62e4d23" is_pre_election: true
I20260929 18:36:13.618986 7975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfa9b6be92465ca74b965a395f2af3" candidate_uuid: "f3bf34bc861148b6be55e008cbd1b809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d1e9c100d4af8aca9ecd25f871875" is_pre_election: true
I20260929 18:36:13.619424 8111 raft_consensus.cc:2454] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bf34bc861148b6be55e008cbd1b809 in term 0.
I20260929 18:36:13.619457 7975 raft_consensus.cc:2454] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bf34bc861148b6be55e008cbd1b809 in term 0.
I20260929 18:36:13.619900 7772 leader_election.cc:302] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc940daaccb4500a084d85cf62e4d23, f3bf34bc861148b6be55e008cbd1b809; no voters:
I20260929 18:36:13.620154 8199 raft_consensus.cc:2781] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:13.620240 8199 raft_consensus.cc:482] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:13.620281 8199 raft_consensus.cc:3037] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:13.621269 8199 raft_consensus.cc:504] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.621590 8199 leader_election.cc:288] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [CANDIDATE]: Term 1 election: Requested vote from peers e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181), bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3:46357)
I20260929 18:36:13.621960 8111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfa9b6be92465ca74b965a395f2af3" candidate_uuid: "f3bf34bc861148b6be55e008cbd1b809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc940daaccb4500a084d85cf62e4d23"
I20260929 18:36:13.621955 7975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfa9b6be92465ca74b965a395f2af3" candidate_uuid: "f3bf34bc861148b6be55e008cbd1b809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d1e9c100d4af8aca9ecd25f871875"
I20260929 18:36:13.622116 8111 raft_consensus.cc:3037] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:13.622113 7975 raft_consensus.cc:3037] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:13.623108 7975 raft_consensus.cc:2454] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bf34bc861148b6be55e008cbd1b809 in term 1.
I20260929 18:36:13.623224 8111 raft_consensus.cc:2454] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bf34bc861148b6be55e008cbd1b809 in term 1.
I20260929 18:36:13.623440 7773 leader_election.cc:302] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e72d1e9c100d4af8aca9ecd25f871875, f3bf34bc861148b6be55e008cbd1b809; no voters:
I20260929 18:36:13.623615 8199 raft_consensus.cc:2781] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:13.623804 8199 raft_consensus.cc:686] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 1 LEADER]: Becoming Leader. State: Replica: f3bf34bc861148b6be55e008cbd1b809, State: Running, Role: LEADER
I20260929 18:36:13.624219 8199 consensus_queue.cc:237] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:13.628117 6921 catalog_manager.cc:5495] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1). New cstate: current_term: 1 leader_uuid: "f3bf34bc861148b6be55e008cbd1b809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } health_report { overall_health: UNKNOWN } } }
I20260929 18:36:13.646950 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.643384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.0.198.1:37347)
0929 18:36:13.643508 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.643512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.643527 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:36:13.643675 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.643678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.643733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.643821 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:13.644076 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.644585 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.645493 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.645704 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.645788 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.646144 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:13.646574 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:13.646683 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.646758 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.646813 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:36:13.652745 7975 raft_consensus.cc:1261] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Refusing update from remote peer f3bf34bc861148b6be55e008cbd1b809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:13.652745 8111 raft_consensus.cc:1261] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 1 FOLLOWER]: Refusing update from remote peer f3bf34bc861148b6be55e008cbd1b809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:13.653412 8199 consensus_queue.cc:1035] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:13.653600 8203 consensus_queue.cc:1035] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:36:13.662361 8160 tablet.cc:2369] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:13.662925 8206 mvcc.cc:205] Tried to move back new op lower bound from 7334735764074991616 to 7334735763967496192. Current Snapshot: MvccSnapshot[applied={T|T < 7334735764074991616}]
I20260929 18:36:13.670400 8208 mvcc.cc:205] Tried to move back new op lower bound from 7334735764074991616 to 7334735763967496192. Current Snapshot: MvccSnapshot[applied={T|T < 7334735764074991616}]
I20260929 18:36:13.672609 6922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:13.672781 6922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:13.677855 6922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc2c11a01b9d447a8c78d40f3d1e9726: Sending DeleteTablet for 3 replicas of tablet 81cfa9b6be92465ca74b965a395f2af3
I20260929 18:36:13.678632 7818 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cfa9b6be92465ca74b965a395f2af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:13 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:49276
I20260929 18:36:13.678648 7952 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cfa9b6be92465ca74b965a395f2af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:13 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33104
I20260929 18:36:13.678836 8091 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cfa9b6be92465ca74b965a395f2af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:13 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:50140
I20260929 18:36:13.679318 8217 tablet_replica.cc:330] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: stopping tablet replica
I20260929 18:36:13.679395 8216 tablet_replica.cc:330] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: stopping tablet replica
I20260929 18:36:13.679735 8216 raft_consensus.cc:2229] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:13.679752 8217 raft_consensus.cc:2229] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:13.680073 8216 raft_consensus.cc:2258] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:13.680224 8217 raft_consensus.cc:2258] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:13.681843 8216 ts_tablet_manager.cc:1905] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:13.681861 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 6890
I20260929 18:36:13.682976 8217 ts_tablet_manager.cc:1905] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:13.685673 8216 ts_tablet_manager.cc:1918] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:13.685667 8217 ts_tablet_manager.cc:1918] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:13.685774 8216 log.cc:1196] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/81cfa9b6be92465ca74b965a395f2af3
I20260929 18:36:13.685899 8217 log.cc:1196] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/81cfa9b6be92465ca74b965a395f2af3
I20260929 18:36:13.686106 8216 ts_tablet_manager.cc:1939] T 81cfa9b6be92465ca74b965a395f2af3 P e72d1e9c100d4af8aca9ecd25f871875: Deleting consensus metadata
I20260929 18:36:13.686373 8217 ts_tablet_manager.cc:1939] T 81cfa9b6be92465ca74b965a395f2af3 P f3bf34bc861148b6be55e008cbd1b809: Deleting consensus metadata
I20260929 18:36:13.687543 8218 tablet_replica.cc:330] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: stopping tablet replica
I20260929 18:36:13.687853 8218 raft_consensus.cc:2229] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:13.688194 8218 raft_consensus.cc:2258] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:36:13.688946 7908 connection.cc:535] server connection from 127.0.0.1:33104 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:36:13.689060 7771 connection.cc:535] server connection from 127.0.0.1:49276 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:36:13.690450 8218 ts_tablet_manager.cc:1905] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:13.692979 8218 ts_tablet_manager.cc:1918] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:13.693135 8218 log.cc:1196] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/81cfa9b6be92465ca74b965a395f2af3
I20260929 18:36:13.693509 8218 ts_tablet_manager.cc:1939] T 81cfa9b6be92465ca74b965a395f2af3 P bcc940daaccb4500a084d85cf62e4d23: Deleting consensus metadata
W20260929 18:36:13.694600 8047 connection.cc:441] server connection from 127.0.0.1:50140 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50140 (request call id 1) could send its response
I20260929 18:36:13.699695 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.681223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.0.198.61:40923)
0929 18:36:13.681337 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.681341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.690291 (+ 8950us) server_negotiation.cc:407] Connection header received
0929 18:36:13.690502 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.690506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.690562 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.690617 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.691645 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.692464 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.693583 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.693949 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.694076 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.696289 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.696293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.696296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.696327 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.697839 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.698351 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.698355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.698357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.698453 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.698758 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.698761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.698763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.698970 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.699070 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.699395 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.699531 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:36:13.704932 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.695035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.0.198.60:40435)
0929 18:36:13.695160 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.695164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.695375 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:36:13.695557 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.695561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.695640 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.695706 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.696434 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.697234 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.698618 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.698941 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.699014 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.700637 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.700641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.700644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.700686 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.702355 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.703251 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.703255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.703257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.703339 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.703727 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.703731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.703732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.703943 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.704044 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.704627 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.704763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:36:13.707542 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.698348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.0.198.61:40923)
0929 18:36:13.698469 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.698473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.698572 (+ 99us) server_negotiation.cc:407] Connection header received
0929 18:36:13.698723 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.698726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.698766 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.698808 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.699586 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.700364 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.701948 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.702312 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.702390 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.703881 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.703901 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.703904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.703926 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.705696 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.706303 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.706306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.706307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.706388 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.706682 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.706685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.706687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.706870 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.706950 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.707286 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.707391 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:36:13.709344 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.680718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.0.198.60:40435)
0929 18:36:13.680824 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:13.680829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:13.700986 (+ 20157us) server_negotiation.cc:407] Connection header received
0929 18:36:13.701133 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:13.701136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:13.701179 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:13.701225 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:13.702005 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.702757 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.703870 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:13.704203 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:13.704270 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:13.705724 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:13.705728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:13.705731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:13.705756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:13.707307 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.707754 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.707757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.707759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.707833 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:13.708462 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:13.708465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:13.708467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:13.708672 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:13.708763 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:13.709135 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 18:36:13.709235 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:36:14.666430 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:14.666020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.2:40513)
0929 18:36:14.666240 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:14.666328 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
W20260929 18:36:14.666760 8022 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:35471 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:14.669854 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:14.669360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.1:39077)
0929 18:36:14.669639 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:14.669749 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
W20260929 18:36:14.670141 7885 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:35471 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:14.670645 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:14.670258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:47973)
0929 18:36:14.670501 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:14.670571 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
W20260929 18:36:14.670907 8158 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:35471 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:15.177846 8229 raft_consensus.cc:482] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc2c11a01b9d447a8c78d40f3d1e9726)
I20260929 18:36:15.178179 8229 raft_consensus.cc:504] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } }
I20260929 18:36:15.178228 8230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc2c11a01b9d447a8c78d40f3d1e9726)
I20260929 18:36:15.178431 8230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } }
I20260929 18:36:15.179207 8229 leader_election.cc:288] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc2c11a01b9d447a8c78d40f3d1e9726 (127.0.198.62:35471), 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60:40435)
I20260929 18:36:15.179864 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.179379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49732)
0929 18:36:15.179683 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.179764 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":55,"threads_started":1}
W20260929 18:36:15.180048 7510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:15.180117 8230 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc2c11a01b9d447a8c78d40f3d1e9726 (127.0.198.62:35471), f22c7915ba60461a9e05af9580de3851 (127.0.198.61:40923)
I20260929 18:36:15.180275 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.179403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49744)
0929 18:36:15.180123 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.180188 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":64,"threads_started":1}
W20260929 18:36:15.180426 7692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:15.180981 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.180716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49746)
0929 18:36:15.180868 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.180915 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":31}
W20260929 18:36:15.181080 7510 leader_election.cc:334] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dc2c11a01b9d447a8c78d40f3d1e9726 (127.0.198.62:35471): Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:15.181301 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.181105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49754)
0929 18:36:15.181198 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.181246 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 18:36:15.181416 7692 leader_election.cc:334] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dc2c11a01b9d447a8c78d40f3d1e9726 (127.0.198.62:35471): Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:15.184152 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.179451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:40435 (local address 127.0.0.1:33940)
0929 18:36:15.179739 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.179752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:15.179855 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:15.180489 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:15.180493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:15.180515 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:15.180631 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.180639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.181808 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.181812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.182965 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.182972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.183805 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.183807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.183887 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.183906 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:36:15.183958 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:36:15.184014 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":67,"threads_started":1}
I20260929 18:36:15.184156 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.179624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.0.198.60:40435)
0929 18:36:15.179815 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.179819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.179836 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:36:15.180099 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.180102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.180278 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.180774 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:15.180876 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.181713 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.183069 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.183707 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.183813 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.183871 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.183970 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.184034 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":125,"spinlock_wait_cycles":1280,"thread_start_us":49,"threads_started":1}
I20260929 18:36:15.184687 7718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f22c7915ba60461a9e05af9580de3851" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" is_pre_election: true
I20260929 18:36:15.184906 7718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22c7915ba60461a9e05af9580de3851 in term 1.
I20260929 18:36:15.185200 7507 leader_election.cc:302] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa8d5aff8a54c7cbf92857373a82daf, f22c7915ba60461a9e05af9580de3851; no voters: dc2c11a01b9d447a8c78d40f3d1e9726
I20260929 18:36:15.185302 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.179969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:40923 (local address 127.0.0.1:47412)
0929 18:36:15.180316 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.180328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:15.180787 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:15.181642 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:15.181646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:15.181665 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:15.181745 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.181751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.182768 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.182770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.184036 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.184044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.184826 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.184830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.185014 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.185038 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:36:15.185116 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:36:15.185184 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":29,"thread_start_us":62,"threads_started":1}
I20260929 18:36:15.185325 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.180403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.0.198.61:40923)
0929 18:36:15.181402 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.181414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.181428 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:15.181470 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.181472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.181513 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.181597 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:15.181832 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.182655 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.184144 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.184715 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.184813 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.184865 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.185142 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.185212 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":682,"thread_start_us":92,"threads_started":1}
I20260929 18:36:15.185593 8229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:36:15.185648 8229 raft_consensus.cc:482] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dc2c11a01b9d447a8c78d40f3d1e9726)
I20260929 18:36:15.185685 8229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:36:15.185729 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f22c7915ba60461a9e05af9580de3851" is_pre_election: true
I20260929 18:36:15.186734 8229 raft_consensus.cc:504] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } }
I20260929 18:36:15.187057 7533 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa8d5aff8a54c7cbf92857373a82daf in current term 2: Already voted for candidate f22c7915ba60461a9e05af9580de3851 in this term.
I20260929 18:36:15.187101 8229 leader_election.cc:288] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [CANDIDATE]: Term 2 election: Requested vote from peers dc2c11a01b9d447a8c78d40f3d1e9726 (127.0.198.62:35471), 4aa8d5aff8a54c7cbf92857373a82daf (127.0.198.60:40435)
I20260929 18:36:15.187417 7718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f22c7915ba60461a9e05af9580de3851" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4aa8d5aff8a54c7cbf92857373a82daf"
I20260929 18:36:15.187537 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.187204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49768)
0929 18:36:15.187428 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.187479 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260929 18:36:15.187539 7718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:36:15.187700 7690 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa8d5aff8a54c7cbf92857373a82daf; no voters: dc2c11a01b9d447a8c78d40f3d1e9726, f22c7915ba60461a9e05af9580de3851
I20260929 18:36:15.188453 7718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22c7915ba60461a9e05af9580de3851 in term 2.
I20260929 18:36:15.188560 8230 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:36:15.188699 7507 leader_election.cc:302] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa8d5aff8a54c7cbf92857373a82daf, f22c7915ba60461a9e05af9580de3851; no voters:
I20260929 18:36:15.188843 8229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:36:15.189152 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.188962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49774)
0929 18:36:15.189061 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.189106 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:36:15.189132 8229 raft_consensus.cc:686] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [term 2 LEADER]: Becoming Leader. State: Replica: f22c7915ba60461a9e05af9580de3851, State: Running, Role: LEADER
W20260929 18:36:15.189252 7510 leader_election.cc:334] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dc2c11a01b9d447a8c78d40f3d1e9726 (127.0.198.62:35471): Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
I20260929 18:36:15.189486 8229 consensus_queue.cc:237] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } }
I20260929 18:36:15.190546 8239 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f22c7915ba60461a9e05af9580de3851. Latest consensus state: current_term: 2 leader_uuid: "f22c7915ba60461a9e05af9580de3851" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:15.190635 8239 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:15.190896 8241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:15.193125 8241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:15.193606 8241 catalog_manager.cc:1259] Loaded cluster ID: 001e9bcc2a3a42b1821a7b5a5f08184c
I20260929 18:36:15.193662 8241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:15.194234 8241 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:15.194578 8241 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Loaded TSK: 0
I20260929 18:36:15.195245 8241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:15.217557 7522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:15.220806 7522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:15.223625 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.223387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49792)
0929 18:36:15.223489 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.223536 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:36:15.223762 7718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [term 2 FOLLOWER]: Refusing update from remote peer f22c7915ba60461a9e05af9580de3851: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 18:36:15.224252 8239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 18:36:15.224334 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.224127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.0.1:49794)
0929 18:36:15.224230 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.224279 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:36:15.224447 7510 consensus_peers.cc:487] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 -> Peer dc2c11a01b9d447a8c78d40f3d1e9726 (127.0.198.62:35471): Couldn't send request to peer dc2c11a01b9d447a8c78d40f3d1e9726. Status: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:36:15.225571 8230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f22c7915ba60461a9e05af9580de3851. Latest consensus state: current_term: 2 leader_uuid: "f22c7915ba60461a9e05af9580de3851" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:15.225708 8230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:36:15.226866 8229 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f22c7915ba60461a9e05af9580de3851" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:15.226997 8229 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:15.227116 8243 mvcc.cc:205] Tried to move back new op lower bound from 7334735770511929344 to 7334735770378735616. Current Snapshot: MvccSnapshot[applied={T|T < 7334735770511929344}]
I20260929 18:36:15.227245 8230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f22c7915ba60461a9e05af9580de3851" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dc2c11a01b9d447a8c78d40f3d1e9726" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35471 } } peers { permanent_uuid: "f22c7915ba60461a9e05af9580de3851" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 40923 } attrs { promote: false } } peers { permanent_uuid: "4aa8d5aff8a54c7cbf92857373a82daf" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 40435 } attrs { promote: false } } }
I20260929 18:36:15.227363 8230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8d5aff8a54c7cbf92857373a82daf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:36:15.236256 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.232437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:40181 (local address 127.0.0.1:33116)
0929 18:36:15.232580 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.232595 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:15.232672 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:15.233053 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:15.233056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:15.233086 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:15.233192 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.233198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.234223 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.234226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.235250 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.235258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.235879 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.235883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.235963 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.235986 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:36:15.236051 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:36:15.236123 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 18:36:15.236265 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.232542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.0.198.2:40181)
0929 18:36:15.232802 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.232807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.232821 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:15.232870 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.232873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.232921 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.233287 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:15.233387 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.234108 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.235367 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.235770 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.235864 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.235929 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.236081 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.236152 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260929 18:36:15.236819 7952 tablet_service.cc:1467] Processing CreateTablet for tablet 20e7bc088f974e3f9dbbba0a3e6f805f (DEFAULT_TABLE table=test-table [id=a320353fb5cb4926a67b4b27f03820cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:15.237105 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e7bc088f974e3f9dbbba0a3e6f805f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:15.237564 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.233340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.0.198.3:46357)
0929 18:36:15.233577 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.233581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.233594 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:15.233638 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.233641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.233681 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.233760 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:15.234036 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.234798 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.236162 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.237071 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.237157 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.237220 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.237375 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.237447 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260929 18:36:15.237558 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.232898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:46357 (local address 127.0.0.1:50148)
0929 18:36:15.233120 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.233133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:15.233196 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:15.233787 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:15.233791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:15.233809 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:15.233927 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.233933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.234913 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.234917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.236029 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.236037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.237170 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.237174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.237269 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.237290 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:36:15.237351 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:36:15.237432 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":5}
I20260929 18:36:15.238090 8091 tablet_service.cc:1467] Processing CreateTablet for tablet 20e7bc088f974e3f9dbbba0a3e6f805f (DEFAULT_TABLE table=test-table [id=a320353fb5cb4926a67b4b27f03820cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:15.238333 8091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e7bc088f974e3f9dbbba0a3e6f805f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:15.238600 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.232941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37347 (local address 127.0.0.1:49298)
0929 18:36:15.233761 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.233773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:15.233857 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:15.234147 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:15.234149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:15.234163 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:15.234251 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.234257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.234963 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.234966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.236843 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.236853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.238199 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.238203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.238287 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.238308 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:36:15.238383 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:36:15.238465 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":33,"spinlock_wait_cycles":1280,"thread_start_us":108,"threads_started":1}
I20260929 18:36:15.238935 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.233386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.0.198.1:37347)
0929 18:36:15.233650 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.233653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.233931 (+ 278us) server_negotiation.cc:407] Connection header received
0929 18:36:15.234011 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.234014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.234064 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.234134 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:15.234344 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.234850 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.237034 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.237946 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.238079 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.238142 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.238448 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.238540 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260929 18:36:15.239459 7818 tablet_service.cc:1467] Processing CreateTablet for tablet 20e7bc088f974e3f9dbbba0a3e6f805f (DEFAULT_TABLE table=test-table [id=a320353fb5cb4926a67b4b27f03820cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:15.239696 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e7bc088f974e3f9dbbba0a3e6f805f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:15.239701 8252 tablet_bootstrap.cc:492] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: Bootstrap starting.
I20260929 18:36:15.240530 8252 tablet_bootstrap.cc:654] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:15.240679 8253 tablet_bootstrap.cc:492] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: Bootstrap starting.
I20260929 18:36:15.241437 8253 tablet_bootstrap.cc:654] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:15.241762 8252 tablet_bootstrap.cc:492] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: No bootstrap required, opened a new log
I20260929 18:36:15.241837 8252 ts_tablet_manager.cc:1397] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:15.241871 8255 tablet_bootstrap.cc:492] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809: Bootstrap starting.
I20260929 18:36:15.242301 8252 raft_consensus.cc:348] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.242401 8252 raft_consensus.cc:374] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:15.242430 8252 raft_consensus.cc:729] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72d1e9c100d4af8aca9ecd25f871875, State: Initialized, Role: FOLLOWER
I20260929 18:36:15.242575 8252 consensus_queue.cc:260] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.242911 8252 ts_tablet_manager.cc:1428] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:36:15.242974 8255 tablet_bootstrap.cc:654] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:15.243189 8253 tablet_bootstrap.cc:492] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: No bootstrap required, opened a new log
I20260929 18:36:15.243314 8253 ts_tablet_manager.cc:1397] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:15.243892 8253 raft_consensus.cc:348] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.244046 8253 raft_consensus.cc:374] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:15.244118 8253 raft_consensus.cc:729] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc940daaccb4500a084d85cf62e4d23, State: Initialized, Role: FOLLOWER
I20260929 18:36:15.244319 8253 consensus_queue.cc:260] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.244393 8255 tablet_bootstrap.cc:492] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809: No bootstrap required, opened a new log
I20260929 18:36:15.244478 8255 ts_tablet_manager.cc:1397] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:36:15.244741 8253 ts_tablet_manager.cc:1428] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:15.244854 8255 raft_consensus.cc:348] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.244962 8255 raft_consensus.cc:374] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:15.244995 8255 raft_consensus.cc:729] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bf34bc861148b6be55e008cbd1b809, State: Initialized, Role: FOLLOWER
I20260929 18:36:15.245103 8255 consensus_queue.cc:260] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.245416 8255 ts_tablet_manager.cc:1428] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:15.246619 8023 heartbeater.cc:503] Master 127.0.198.61:40923 was elected leader, sending a full tablet report...
I20260929 18:36:15.247241 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.245667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:55981)
0929 18:36:15.247114 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.247166 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1384,"thread_start_us":104,"threads_started":1}
I20260929 18:36:15.247260 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.246263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.1:45981)
0929 18:36:15.247114 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.247169 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":96,"threads_started":1}
I20260929 18:36:15.247967 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.247746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:49953)
0929 18:36:15.247862 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.247907 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:36:15.248199 8159 heartbeater.cc:503] Master 127.0.198.61:40923 was elected leader, sending a full tablet report...
I20260929 18:36:15.248580 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.248151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.1:46063)
0929 18:36:15.248475 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.248522 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260929 18:36:15.248849 7886 heartbeater.cc:503] Master 127.0.198.61:40923 was elected leader, sending a full tablet report...
I20260929 18:36:15.249059 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.243628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.2:52075)
0929 18:36:15.248907 (+ 5279us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.248961 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5216,"thread_start_us":85,"threads_started":1}
I20260929 18:36:15.249744 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.249423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.2:34175)
0929 18:36:15.249634 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.249681 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260929 18:36:15.382010 8260 raft_consensus.cc:482] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:15.382175 8260 raft_consensus.cc:504] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.383265 8260 leader_election.cc:288] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181), f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1:37347)
I20260929 18:36:15.387704 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.383515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37347 (local address 127.0.198.3:56003)
0929 18:36:15.383724 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.383738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:15.383875 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:15.384596 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:15.384599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:15.384617 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:15.384724 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.384731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.385585 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.385588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.386363 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.386373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.387230 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.387233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.387320 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.387350 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:36:15.387408 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:36:15.387525 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":44}
I20260929 18:36:15.387996 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.383911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:52185 (local address 127.0.198.2:40181)
0929 18:36:15.384014 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.384018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.384030 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:15.384136 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.384139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.384182 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.384242 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:15.384474 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.385151 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.386392 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.386992 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.387102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.387167 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.387546 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.387743 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:36:15.387713 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.383539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:40181 (local address 127.0.198.3:52185)
0929 18:36:15.383916 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.383929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:15.384010 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:15.384262 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:15.384264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:15.384278 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:15.384373 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.384379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.385253 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.385256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.386266 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:15.386275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.387165 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.387170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:15.387289 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.387332 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:36:15.387408 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:36:15.387525 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":4,"thread_start_us":120,"threads_started":1}
I20260929 18:36:15.388270 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.383620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:56003 (local address 127.0.198.1:37347)
0929 18:36:15.383725 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.383729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.383800 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:36:15.384392 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.384395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.384460 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.384531 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:15.384857 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.385466 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.386485 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.387069 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.387177 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.387243 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.387469 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.387667 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:36:15.390478 7975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e7bc088f974e3f9dbbba0a3e6f805f" candidate_uuid: "bcc940daaccb4500a084d85cf62e4d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d1e9c100d4af8aca9ecd25f871875" is_pre_election: true
I20260929 18:36:15.390522 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e7bc088f974e3f9dbbba0a3e6f805f" candidate_uuid: "bcc940daaccb4500a084d85cf62e4d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bf34bc861148b6be55e008cbd1b809" is_pre_election: true
I20260929 18:36:15.390697 7975 raft_consensus.cc:2454] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc940daaccb4500a084d85cf62e4d23 in term 0.
I20260929 18:36:15.390738 7838 raft_consensus.cc:2454] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc940daaccb4500a084d85cf62e4d23 in term 0.
I20260929 18:36:15.391168 8046 leader_election.cc:302] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc940daaccb4500a084d85cf62e4d23, e72d1e9c100d4af8aca9ecd25f871875; no voters:
I20260929 18:36:15.391443 8260 raft_consensus.cc:2781] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:15.391567 8260 raft_consensus.cc:482] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:15.391645 8260 raft_consensus.cc:3037] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:15.392771 8260 raft_consensus.cc:504] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.393149 8260 leader_election.cc:288] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [CANDIDATE]: Term 1 election: Requested vote from peers e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181), f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1:37347)
I20260929 18:36:15.393510 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e7bc088f974e3f9dbbba0a3e6f805f" candidate_uuid: "bcc940daaccb4500a084d85cf62e4d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bf34bc861148b6be55e008cbd1b809"
I20260929 18:36:15.393510 7975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e7bc088f974e3f9dbbba0a3e6f805f" candidate_uuid: "bcc940daaccb4500a084d85cf62e4d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d1e9c100d4af8aca9ecd25f871875"
I20260929 18:36:15.393636 7975 raft_consensus.cc:3037] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:15.393636 7838 raft_consensus.cc:3037] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:15.394759 7975 raft_consensus.cc:2454] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc940daaccb4500a084d85cf62e4d23 in term 1.
I20260929 18:36:15.394842 7838 raft_consensus.cc:2454] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc940daaccb4500a084d85cf62e4d23 in term 1.
I20260929 18:36:15.395095 8047 leader_election.cc:302] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc940daaccb4500a084d85cf62e4d23, f3bf34bc861148b6be55e008cbd1b809; no voters:
I20260929 18:36:15.395254 8260 raft_consensus.cc:2781] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:15.395471 8260 raft_consensus.cc:686] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 1 LEADER]: Becoming Leader. State: Replica: bcc940daaccb4500a084d85cf62e4d23, State: Running, Role: LEADER
I20260929 18:36:15.395900 8260 consensus_queue.cc:237] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } }
I20260929 18:36:15.396130 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.395884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:37691)
0929 18:36:15.395999 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.396059 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:36:15.398311 7522 catalog_manager.cc:5495] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3). New cstate: current_term: 1 leader_uuid: "bcc940daaccb4500a084d85cf62e4d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcc940daaccb4500a084d85cf62e4d23" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46357 } health_report { overall_health: HEALTHY } } }
I20260929 18:36:15.428884 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.425323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.0.198.3:46357)
0929 18:36:15.425433 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:15.425436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:15.425449 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:15.425553 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:15.425556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:15.425614 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:15.425707 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:15.425925 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.426496 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.427442 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:15.427651 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:15.427728 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:15.428014 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:15.428480 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:15.428567 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:15.428715 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:36:15.428766 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:36:15.431818 7975 raft_consensus.cc:1261] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Refusing update from remote peer bcc940daaccb4500a084d85cf62e4d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:15.432080 7838 raft_consensus.cc:1261] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 1 FOLLOWER]: Refusing update from remote peer bcc940daaccb4500a084d85cf62e4d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:15.432425 8266 consensus_queue.cc:1035] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72d1e9c100d4af8aca9ecd25f871875" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:15.432634 8260 consensus_queue.cc:1035] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3bf34bc861148b6be55e008cbd1b809" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:15.434862 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.434414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.1:51779)
0929 18:36:15.434710 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.434776 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
I20260929 18:36:15.435019 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.434743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.2:55975)
0929 18:36:15.434891 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.434941 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 18:36:15.438501 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.437928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.2:57433)
0929 18:36:15.438362 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.438419 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362}
I20260929 18:36:15.438891 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.438273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:41661)
0929 18:36:15.438779 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.438826 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443}
I20260929 18:36:15.438894 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.438633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.1:37105)
0929 18:36:15.438770 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.438819 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 18:36:15.439786 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.439550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35471 (local address 127.0.198.3:59103)
0929 18:36:15.439679 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.439724 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 18:36:15.448289 7522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:15.448488 7522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:15.452790 7522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f22c7915ba60461a9e05af9580de3851: Sending DeleteTablet for 3 replicas of tablet 20e7bc088f974e3f9dbbba0a3e6f805f
I20260929 18:36:15.453496 7818 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e7bc088f974e3f9dbbba0a3e6f805f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:15 UTC) from {username='oryx', principal='oryx/127.0.198.61@KRBTEST.COM'} at 127.0.0.1:49298
I20260929 18:36:15.453569 7952 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e7bc088f974e3f9dbbba0a3e6f805f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:15 UTC) from {username='oryx', principal='oryx/127.0.198.61@KRBTEST.COM'} at 127.0.0.1:33116
I20260929 18:36:15.453876 8278 tablet_replica.cc:330] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809: stopping tablet replica
I20260929 18:36:15.454042 8278 raft_consensus.cc:2229] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:15.454092 8091 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e7bc088f974e3f9dbbba0a3e6f805f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:15 UTC) from {username='oryx', principal='oryx/127.0.198.61@KRBTEST.COM'} at 127.0.0.1:50148
I20260929 18:36:15.454195 8278 raft_consensus.cc:2258] T 20e7bc088f974e3f9dbbba0a3e6f805f P f3bf34bc861148b6be55e008cbd1b809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:15.454442 8280 tablet_replica.cc:330] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: stopping tablet replica
I20260929 18:36:15.454473 8279 tablet_replica.cc:330] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: stopping tablet replica
I20260929 18:36:15.454558 8280 raft_consensus.cc:2229] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:15.454607 8279 raft_consensus.cc:2229] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:15.454756 8279 raft_consensus.cc:2258] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:15.454895 8280 raft_consensus.cc:2258] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:15.455147 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 7667
I20260929 18:36:15.456741 8280 ts_tablet_manager.cc:1905] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:15.457028 8279 ts_tablet_manager.cc:1905] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:15.459595 8280 ts_tablet_manager.cc:1918] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:15.459610 8279 ts_tablet_manager.cc:1918] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:15.459684 8279 log.cc:1196] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/20e7bc088f974e3f9dbbba0a3e6f805f
I20260929 18:36:15.459683 8280 log.cc:1196] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/20e7bc088f974e3f9dbbba0a3e6f805f
I20260929 18:36:15.459951 8279 ts_tablet_manager.cc:1939] T 20e7bc088f974e3f9dbbba0a3e6f805f P e72d1e9c100d4af8aca9ecd25f871875: Deleting consensus metadata
I20260929 18:36:15.459972 8280 ts_tablet_manager.cc:1939] T 20e7bc088f974e3f9dbbba0a3e6f805f P bcc940daaccb4500a084d85cf62e4d23: Deleting consensus metadata
I20260929 18:36:15.460887 7509 catalog_manager.cc:4841] TS e72d1e9c100d4af8aca9ecd25f871875 (127.0.198.2:40181): tablet 20e7bc088f974e3f9dbbba0a3e6f805f (table test-table [id=a320353fb5cb4926a67b4b27f03820cc]) successfully deleted
I20260929 18:36:15.460889 7508 catalog_manager.cc:4841] TS bcc940daaccb4500a084d85cf62e4d23 (127.0.198.3:46357): tablet 20e7bc088f974e3f9dbbba0a3e6f805f (table test-table [id=a320353fb5cb4926a67b4b27f03820cc]) successfully deleted
I20260929 18:36:15.463526 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 7892
I20260929 18:36:15.464231 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.463622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37347 (local address 127.0.0.1:49312)
0929 18:36:15.464097 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.464155 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
W20260929 18:36:15.464421 7510 catalog_manager.cc:4584] TS f3bf34bc861148b6be55e008cbd1b809 (127.0.198.1:37347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20e7bc088f974e3f9dbbba0a3e6f805f: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37347: connect: Connection refused (error 111)
I20260929 18:36:15.468874 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 8028
I20260929 18:36:15.474162 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 7386
I20260929 18:36:15.479486 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 7495
2026-09-29T18:36:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5289 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:15 dist-test-slave-q5qx krb5kdc[8283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:15 dist-test-slave-q5qx krb5kdc[8283](info): set up 2 sockets
Sep 29 18:36:15 dist-test-slave-q5qx krb5kdc[8283](info): commencing operation
krb5kdc: starting...
W20260929 18:36:16.438618 792 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:16 dist-test-slave-q5qx krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab.
Entry for principal oryx/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab.
I20260929 18:36:16.559049 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38883
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab
--principal=oryx/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:38883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:16.668838 8296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:16.669231 8296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:16.669303 8296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:16.673609 8296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:16.673684 8296 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:36:16.673700 8296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:16.673717 8296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:16.673732 8296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:16.673746 8296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:16.679234 8296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:38883
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.62.keytab
--principal=oryx/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:16.680305 8296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:16.681306 8296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:16.689945 8296 server_base.cc:1034] running on GCE node
W20260929 18:36:16.689911 8302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:16.689911 8301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:16.690104 8304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:16.690701 8296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:16.691875 8296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:16.693069 8296 hybrid_clock.cc:648] HybridClock initialized: now 1790706976693061 us; error 31 us; skew 500 ppm
Sep 29 18:36:16 dist-test-slave-q5qx krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:16.704239 8296 init.cc:377] Logged in from keytab as oryx/127.0.198.62@KRBTEST.COM (short username oryx)
I20260929 18:36:16.705319 8296 webserver.cc:460] Webserver started at http://127.0.198.62:43783/ using document root <none> and password file <none>
I20260929 18:36:16.705917 8296 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:16.706009 8296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:16.706211 8296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:16.708349 8296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e"
format_stamp: "Formatted at 2026-09-29 18:36:16 on dist-test-slave-q5qx"
server_key: "2863d69442b2e54c063971f0a939c7b0"
server_key_iv: "07b7cadc53653225cd770c82e313e441"
server_key_version: "encryptionkey@0"
I20260929 18:36:16.708874 8296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e"
format_stamp: "Formatted at 2026-09-29 18:36:16 on dist-test-slave-q5qx"
server_key: "2863d69442b2e54c063971f0a939c7b0"
server_key_iv: "07b7cadc53653225cd770c82e313e441"
server_key_version: "encryptionkey@0"
I20260929 18:36:16.712832 8296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:16.715255 8311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:16.716571 8296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:16.716675 8296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e"
format_stamp: "Formatted at 2026-09-29 18:36:16 on dist-test-slave-q5qx"
server_key: "2863d69442b2e54c063971f0a939c7b0"
server_key_iv: "07b7cadc53653225cd770c82e313e441"
server_key_version: "encryptionkey@0"
I20260929 18:36:16.716770 8296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:16.731639 8296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:16.734983 8296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:16.735137 8296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:16.742626 8296 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:38883
I20260929 18:36:16.742650 8373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:38883 every 8 connection(s)
I20260929 18:36:16.743556 8296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:16.745595 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 8296
I20260929 18:36:16.745707 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:16.746037 792 external_mini_cluster.cc:1428] Setting key 0249fcbe6898cf662c135bda8313ed9a
I20260929 18:36:16.746376 8374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:16.753305 8374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Bootstrap starting.
Sep 29 18:36:16 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:16.755724 8374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:16.757019 8374 log.cc:824] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:16.759171 8374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: No bootstrap required, opened a new log
I20260929 18:36:16.760490 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:16.747417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.198.62:38883)
0929 18:36:16.747852 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:16.747869 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:16.747923 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:36:16.748618 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:16.748642 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:16.749073 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:16.749443 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:16.750117 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:16.751556 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:16.752464 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:16.753134 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:16.753283 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:16.755804 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:16.755810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:16.755826 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:16.755857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:16.757660 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:16.758063 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:16.758068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:16.758074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:16.758209 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:16.758433 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:16.758436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:16.758438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:16.758914 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:16.759089 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:16.759294 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:36:16.759579 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":124,"threads_started":1}
I20260929 18:36:16.762755 8374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38883 } }
I20260929 18:36:16.763005 8374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:16.763057 8374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac4cceb12ca4ba5841ea4cb85bf8c3e, State: Initialized, Role: FOLLOWER
I20260929 18:36:16.763444 8374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38883 } }
I20260929 18:36:16.763562 8374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:16.763607 8374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:16.763674 8374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:16.764478 8374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38883 } }
I20260929 18:36:16.764755 8374 leader_election.cc:302] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ac4cceb12ca4ba5841ea4cb85bf8c3e; no voters:
I20260929 18:36:16.765069 8374 leader_election.cc:288] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:16.765187 8379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:16.765404 8379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [term 1 LEADER]: Becoming Leader. State: Replica: 2ac4cceb12ca4ba5841ea4cb85bf8c3e, State: Running, Role: LEADER
I20260929 18:36:16.765720 8379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38883 } }
I20260929 18:36:16.766260 8374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:16.766816 8380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ac4cceb12ca4ba5841ea4cb85bf8c3e. Latest consensus state: current_term: 1 leader_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38883 } } }
I20260929 18:36:16.767033 8380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:16.767272 8381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac4cceb12ca4ba5841ea4cb85bf8c3e" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38883 } } }
I20260929 18:36:16.767370 8381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:16.767524 8388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:16.770150 8388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:16.775303 8388 catalog_manager.cc:1347] Generated new cluster ID: 0fbe3359b35744c0a5b9a7b6c8255ab2
I20260929 18:36:16.775360 8388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:16.783584 8388 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:16.784606 8388 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:16.791623 8388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Generated new TSK 0
I20260929 18:36:16.792451 8388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab.
Entry for principal oryx/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab.
I20260929 18:36:16.826186 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38883
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab
--principal=oryx/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:16.940141 8400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:16.940413 8400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:16.940456 8400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:16.944500 8400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:16.944566 8400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:16.944605 8400 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:36:16.944638 8400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:16.949927 8400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab
--principal=oryx/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:38883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:16.951186 8400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:16.952224 8400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:16.961751 8408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:16.961861 8400 server_base.cc:1034] running on GCE node
W20260929 18:36:16.961753 8406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:16.961750 8405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:16.962455 8400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:16.963065 8400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:16.964269 8400 hybrid_clock.cc:648] HybridClock initialized: now 1790706976964248 us; error 43 us; skew 500 ppm
Sep 29 18:36:16 dist-test-slave-q5qx krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:16.967413 8400 init.cc:377] Logged in from keytab as oryx/127.0.198.1@KRBTEST.COM (short username oryx)
I20260929 18:36:16.968371 8400 webserver.cc:460] Webserver started at http://127.0.198.1:36379/ using document root <none> and password file <none>
I20260929 18:36:16.968911 8400 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:16.968971 8400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:16.969167 8400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:16.971238 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "360e4436f64d4143b1fb23fe26ce0e8d"
format_stamp: "Formatted at 2026-09-29 18:36:16 on dist-test-slave-q5qx"
server_key: "71ce384e364a3e02f0fb221b28940467"
server_key_iv: "871ba3aeaae0893b156e37daecbc50b7"
server_key_version: "encryptionkey@0"
I20260929 18:36:16.971786 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "360e4436f64d4143b1fb23fe26ce0e8d"
format_stamp: "Formatted at 2026-09-29 18:36:16 on dist-test-slave-q5qx"
server_key: "71ce384e364a3e02f0fb221b28940467"
server_key_iv: "871ba3aeaae0893b156e37daecbc50b7"
server_key_version: "encryptionkey@0"
I20260929 18:36:16.975587 8400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:36:16.978173 8415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:16.979547 8400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:16.979650 8400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "360e4436f64d4143b1fb23fe26ce0e8d"
format_stamp: "Formatted at 2026-09-29 18:36:16 on dist-test-slave-q5qx"
server_key: "71ce384e364a3e02f0fb221b28940467"
server_key_iv: "871ba3aeaae0893b156e37daecbc50b7"
server_key_version: "encryptionkey@0"
I20260929 18:36:16.979764 8400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:16.993157 8400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:16.996323 8400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:16.996505 8400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:16.998001 8400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:16.998056 8400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:16.998100 8400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:16.998131 8400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:17.007969 8400 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:39235
I20260929 18:36:17.008056 8530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:39235 every 8 connection(s)
I20260929 18:36:17.009063 8400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:17.011782 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:16.998880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:47549 (local address 127.0.198.62:38883)
0929 18:36:16.999028 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:16.999034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:16.999920 (+ 886us) server_negotiation.cc:407] Connection header received
0929 18:36:17.001113 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.001117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.001178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.001279 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.002908 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.003441 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.004567 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.004812 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.004916 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.008198 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:17.008203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:17.008206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:17.008236 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:17.009742 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.010294 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.010326 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.010327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.010409 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.010684 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.010686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.010687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.010877 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:17.010951 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.011530 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.011654 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:36:17.012668 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:16.999246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38883 (local address 127.0.198.1:47549)
0929 18:36:16.999737 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:16.999796 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.000875 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.001498 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.001515 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.002020 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:17.002741 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.002762 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.003566 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.003569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.004452 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.004461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.004967 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.004972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.005144 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.006349 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:17.006369 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:17.007976 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:17.009867 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.009881 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.009895 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.010198 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.010486 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.010488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.010489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.010607 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.011007 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:17.011013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:17.011264 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.011428 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.011750 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
I20260929 18:36:17.012961 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 8400
I20260929 18:36:17.013075 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:17.013372 792 external_mini_cluster.cc:1428] Setting key 5be412641c601428dad1083102be2e4d
I20260929 18:36:17.014101 8531 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38883
I20260929 18:36:17.014245 8531 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:17.014773 8531 heartbeater.cc:511] Master 127.0.198.62:38883 requested a full tablet report, sending...
I20260929 18:36:17.016170 8328 ts_manager.cc:194] Registered new tserver with Master: 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235)
I20260929 18:36:17.017501 8328 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:47549
WARNING: no policy specified for oryx/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:36:17.023855 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.016308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38883 (local address 127.0.198.1:45777)
0929 18:36:17.016687 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.016701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.016798 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.017025 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.017028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.017235 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:17.017358 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.017366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.018311 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.018315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.019176 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.019188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.019775 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.019779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.019876 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.020432 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:17.020445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:17.020811 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:17.022505 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.022508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.022511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.022707 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.022989 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.022992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.022994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.023069 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.023397 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:17.023399 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:17.023477 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.023556 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.023672 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":101,"threads_started":1}
I20260929 18:36:17.024039 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.016461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:45777 (local address 127.0.198.62:38883)
0929 18:36:17.016569 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.016574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.016770 (+ 196us) server_negotiation.cc:407] Connection header received
0929 18:36:17.016871 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.016874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.016917 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.016998 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.017467 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.018190 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.019309 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.019655 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.019751 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.021008 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:17.021012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:17.021015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:17.021040 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:17.022407 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.022793 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.022797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.022799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.022887 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.023142 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.023145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.023146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.023321 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:17.023403 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.023655 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.023881 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab.
Entry for principal oryx/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab.
I20260929 18:36:17.041755 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38883
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab
--principal=oryx/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:17.151672 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:17.151974 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:17.152019 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:17.156100 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:17.156167 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:17.156204 8540 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:36:17.156235 8540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:17.161648 8540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab
--principal=oryx/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:38883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:17.162703 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:17.163702 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:17.173782 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:17.173785 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:17.173801 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:17.173897 8540 server_base.cc:1034] running on GCE node
I20260929 18:36:17.174378 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:17.175027 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:17.176218 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790706977176197 us; error 33 us; skew 500 ppm
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:17.179118 8540 init.cc:377] Logged in from keytab as oryx/127.0.198.2@KRBTEST.COM (short username oryx)
I20260929 18:36:17.179980 8540 webserver.cc:460] Webserver started at http://127.0.198.2:37295/ using document root <none> and password file <none>
I20260929 18:36:17.180505 8540 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:17.180568 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:17.180755 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:17.182967 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "91f661e2f82c4ae6aeda423adbf82684"
format_stamp: "Formatted at 2026-09-29 18:36:17 on dist-test-slave-q5qx"
server_key: "e3d01d930845dc4dff883f9203595ff4"
server_key_iv: "aaa15c6d722faa530059d054df8cf93c"
server_key_version: "encryptionkey@0"
I20260929 18:36:17.183516 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "91f661e2f82c4ae6aeda423adbf82684"
format_stamp: "Formatted at 2026-09-29 18:36:17 on dist-test-slave-q5qx"
server_key: "e3d01d930845dc4dff883f9203595ff4"
server_key_iv: "aaa15c6d722faa530059d054df8cf93c"
server_key_version: "encryptionkey@0"
I20260929 18:36:17.187175 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 18:36:17.189594 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:17.190973 8540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:36:17.191102 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "91f661e2f82c4ae6aeda423adbf82684"
format_stamp: "Formatted at 2026-09-29 18:36:17 on dist-test-slave-q5qx"
server_key: "e3d01d930845dc4dff883f9203595ff4"
server_key_iv: "aaa15c6d722faa530059d054df8cf93c"
server_key_version: "encryptionkey@0"
I20260929 18:36:17.191205 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:17.217564 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:17.220942 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:17.221115 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:17.222671 8540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:17.222733 8540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:17.222786 8540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:17.222812 8540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:17.232944 8540 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:44965
I20260929 18:36:17.233027 8670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:44965 every 8 connection(s)
I20260929 18:36:17.234076 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:17.236038 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.223442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:33631 (local address 127.0.198.62:38883)
0929 18:36:17.223561 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.223565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.224585 (+ 1020us) server_negotiation.cc:407] Connection header received
0929 18:36:17.225588 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.225592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.225649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.225704 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.227218 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.227835 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.229025 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.229377 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.229496 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.232349 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:17.232353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:17.232356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:17.232388 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:17.233941 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.234483 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.234486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.234488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.234581 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.234968 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.234971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.234973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.235199 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:17.235300 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.235808 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.235922 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:36:17.236793 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.223951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38883 (local address 127.0.198.2:33631)
0929 18:36:17.224419 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.224469 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.225339 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.226169 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.226176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.226506 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:17.227068 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.227083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.227986 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.227989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.228912 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.228922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.229548 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.229553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.229746 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.230346 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:17.230366 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:17.232172 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:17.234072 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.234077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.234088 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.234367 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.234680 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.234683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.234685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.234858 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.235305 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:17.235310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:17.235560 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.235724 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.235981 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260929 18:36:17.238004 8671 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38883
I20260929 18:36:17.238145 8671 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:17.238236 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 8540
I20260929 18:36:17.238363 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:17.238636 792 external_mini_cluster.cc:1428] Setting key c9fa37b9226ff667d5a215b8297375de
I20260929 18:36:17.238732 8671 heartbeater.cc:511] Master 127.0.198.62:38883 requested a full tablet report, sending...
I20260929 18:36:17.239748 8328 ts_manager.cc:194] Registered new tserver with Master: 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965)
I20260929 18:36:17.240422 8328 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:33631
WARNING: no policy specified for oryx/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:36:17.248806 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.240237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:47013 (local address 127.0.198.62:38883)
0929 18:36:17.241897 (+ 1660us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.241901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.241914 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:17.241946 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.241948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.242033 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.242107 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.242538 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.243050 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.244116 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.244331 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.244410 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.245630 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:17.245633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:17.245635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:17.245657 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:17.247086 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.247568 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.247571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.247572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.247652 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.247955 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.247958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.247959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.248164 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:17.248243 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.248538 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.248632 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583}
I20260929 18:36:17.248816 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.240200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38883 (local address 127.0.198.2:47013)
0929 18:36:17.240482 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.240494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.240566 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.242122 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.242125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.242316 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:17.242433 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.242439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.243167 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.243170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.243999 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.244010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.244438 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.244442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.244542 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.245013 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:17.245027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:17.245476 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:17.247195 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.247199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.247201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.247451 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.247746 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.247749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.247752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.247859 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.248252 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:17.248255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:17.248362 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.248471 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.248619 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab.
Entry for principal oryx/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab.
I20260929 18:36:17.266969 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38883
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab
--principal=oryx/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:17.376518 8680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:17.376775 8680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:17.376819 8680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:17.380901 8680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:17.380970 8680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:17.381011 8680 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:36:17.381042 8680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:17.386482 8680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab
--principal=oryx/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.198.62:38883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:17.387852 8680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:17.389194 8680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:17.400283 8688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:17.400290 8686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:17.400309 8685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:17.400552 8680 server_base.cc:1034] running on GCE node
I20260929 18:36:17.401051 8680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:17.401717 8680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:17.402895 8680 hybrid_clock.cc:648] HybridClock initialized: now 1790706977402845 us; error 68 us; skew 500 ppm
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:17.405699 8680 init.cc:377] Logged in from keytab as oryx/127.0.198.3@KRBTEST.COM (short username oryx)
I20260929 18:36:17.406692 8680 webserver.cc:460] Webserver started at http://127.0.198.3:38061/ using document root <none> and password file <none>
I20260929 18:36:17.407258 8680 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:17.407341 8680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:17.407549 8680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:17.409574 8680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "90486608c3ef4ead8332f497801b318d"
format_stamp: "Formatted at 2026-09-29 18:36:17 on dist-test-slave-q5qx"
server_key: "4c73455e5b748a14c2c069a1c791bddd"
server_key_iv: "a48ba23283a2d2eba8f77a7bdfa43c32"
server_key_version: "encryptionkey@0"
I20260929 18:36:17.410143 8680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "90486608c3ef4ead8332f497801b318d"
format_stamp: "Formatted at 2026-09-29 18:36:17 on dist-test-slave-q5qx"
server_key: "4c73455e5b748a14c2c069a1c791bddd"
server_key_iv: "a48ba23283a2d2eba8f77a7bdfa43c32"
server_key_version: "encryptionkey@0"
I20260929 18:36:17.413923 8680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:17.416491 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:17.417724 8680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:17.417834 8680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "90486608c3ef4ead8332f497801b318d"
format_stamp: "Formatted at 2026-09-29 18:36:17 on dist-test-slave-q5qx"
server_key: "4c73455e5b748a14c2c069a1c791bddd"
server_key_iv: "a48ba23283a2d2eba8f77a7bdfa43c32"
server_key_version: "encryptionkey@0"
I20260929 18:36:17.417943 8680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:17.432772 8680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:17.435930 8680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:17.436112 8680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:17.437567 8680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:17.437618 8680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:17.437670 8680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:17.437700 8680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:17.447695 8680 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:41059
I20260929 18:36:17.448069 8810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:41059 every 8 connection(s)
I20260929 18:36:17.448678 8680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:17.450798 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.438339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:34761 (local address 127.0.198.62:38883)
0929 18:36:17.438476 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.438481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.439404 (+ 923us) server_negotiation.cc:407] Connection header received
0929 18:36:17.440449 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.440453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.440513 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.440611 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.442071 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.442617 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.443834 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.444196 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.444320 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.447116 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:17.447120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:17.447123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:17.447155 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:17.448675 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.449282 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.449285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.449287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.449375 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.449728 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.449732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.449733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.449926 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:17.450044 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.450517 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.450639 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:36:17.451602 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.438785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38883 (local address 127.0.198.3:34761)
0929 18:36:17.439241 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.439291 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.440209 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.440878 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.440888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.441290 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:17.441892 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.441906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.442729 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.442733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.443718 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.443728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.444379 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.444383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.444582 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.445188 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:17.445206 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:17.446914 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:17.448793 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.448799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.448813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.449163 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.449461 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.449464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.449466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.449641 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.450037 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:17.450042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:17.450289 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.450455 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.450771 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
I20260929 18:36:17.452750 8811 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38883
I20260929 18:36:17.452903 8811 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:17.453004 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 8680
I20260929 18:36:17.453141 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:17.453411 8811 heartbeater.cc:511] Master 127.0.198.62:38883 requested a full tablet report, sending...
I20260929 18:36:17.453459 792 external_mini_cluster.cc:1428] Setting key 66596f74715ea03ee8ea438bedbb97f7
I20260929 18:36:17.454545 8327 ts_manager.cc:194] Registered new tserver with Master: 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059)
I20260929 18:36:17.455219 8327 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:34761
I20260929 18:36:17.455639 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:17.463599 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.455290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:47879 (local address 127.0.198.62:38883)
0929 18:36:17.455398 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.455402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.455557 (+ 155us) server_negotiation.cc:407] Connection header received
0929 18:36:17.455665 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.455668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.455713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.455781 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.457472 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.457997 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.458954 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.459305 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.459419 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.460539 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:17.460543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:17.460546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:17.460569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:17.462093 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.462500 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.462504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.462505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.462594 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.462904 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.462906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.462907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.463071 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:17.463143 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.463388 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.463479 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:36:17.463675 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.455177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38883 (local address 127.0.198.3:47879)
0929 18:36:17.455470 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.455484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.455586 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.457006 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.457008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.457236 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:17.457353 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.457361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.458109 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.458118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.458768 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.458777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.459437 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.459440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.459520 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.460044 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:17.460055 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:17.460383 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:17.462211 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.462214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.462215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.462407 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.462682 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:17.462685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:17.462687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:17.462784 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:17.463151 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:17.463154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:17.463251 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.463350 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.463489 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706977, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:36:17.479911 8377 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:17.473761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.198.62:38883)
0929 18:36:17.473908 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.473912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.473926 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:17.473965 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.474017 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.474064 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.474135 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.474871 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.475382 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.476279 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.476494 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.476576 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.479714 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46464: BlockingRecv error: recv got EOF from 127.0.0.1:46464 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 18:36:17 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706977, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:17.488822 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.481259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.198.62:38883)
0929 18:36:17.481356 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.481360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.481378 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:36:17.481476 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.481479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.481515 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.481582 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:17.482145 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.482638 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.483461 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.483691 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.483777 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.486116 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:17.486120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:17.486122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:17.486150 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:17.487377 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.487772 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.487774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.487775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.487848 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:17.488098 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:17.488101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:17.488102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:17.488295 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:17.488404 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.488581 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.488671 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:36:17.491715 8327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:17.494730 8327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:17.508237 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.502721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.0.198.3:41059)
0929 18:36:17.503699 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.503711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.503735 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:36:17.503808 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.503813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.504012 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.504516 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:17.504611 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.505323 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.506823 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.507626 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.507733 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.507893 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.508047 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.508120 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":115,"threads_started":1}
I20260929 18:36:17.508272 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.502594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:41059 (local address 127.0.0.1:57112)
0929 18:36:17.503020 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.503070 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.503193 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.504198 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.504205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.504232 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:17.504396 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.504404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.505433 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.505437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.506719 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.506727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.507745 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.507749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.507868 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.507908 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.508015 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.508119 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":55,"thread_start_us":72,"threads_started":1}
I20260929 18:36:17.508919 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.502687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:39235 (local address 127.0.0.1:33410)
0929 18:36:17.503395 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.503408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.503492 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.504216 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.504219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.504245 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:17.504372 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.504377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.505416 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.505420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.506781 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.506790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.508538 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.508542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.508625 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.508649 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.508723 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.508785 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":68,"spinlock_wait_cycles":8448,"thread_start_us":89,"threads_started":1}
I20260929 18:36:17.508996 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.503100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.0.198.1:39235)
0929 18:36:17.503718 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.503734 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.503757 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:36:17.503833 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.503839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.504036 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.504489 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:17.504589 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.505302 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.506883 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.507632 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.507733 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.507893 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.508767 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.508865 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":65,"threads_started":1}
I20260929 18:36:17.509948 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.503617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44965 (local address 127.0.0.1:43390)
0929 18:36:17.504131 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.504143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.504215 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.506156 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.506159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.506187 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:17.506279 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.506284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.507375 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.507378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.508284 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.508293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.509571 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.509575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.509668 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.509702 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.509770 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.509832 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":188,"threads_started":1}
I20260929 18:36:17.510460 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.504112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.198.2:44965)
0929 18:36:17.505561 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.505572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.505594 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:36:17.505663 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.505669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.505850 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.506021 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:17.506393 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.507272 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.508413 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.509226 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.509317 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.509511 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.510219 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.510315 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":88,"threads_started":1}
I20260929 18:36:17.511314 8745 tablet_service.cc:1467] Processing CreateTablet for tablet 880c50b3d0bc44cead618b1eb7febdd6 (DEFAULT_TABLE table=test-table [id=759d451dd56248a0994bcd63b0393a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:17.511653 8464 tablet_service.cc:1467] Processing CreateTablet for tablet 880c50b3d0bc44cead618b1eb7febdd6 (DEFAULT_TABLE table=test-table [id=759d451dd56248a0994bcd63b0393a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:17.512197 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880c50b3d0bc44cead618b1eb7febdd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:17.512661 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880c50b3d0bc44cead618b1eb7febdd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:17.513075 8605 tablet_service.cc:1467] Processing CreateTablet for tablet 880c50b3d0bc44cead618b1eb7febdd6 (DEFAULT_TABLE table=test-table [id=759d451dd56248a0994bcd63b0393a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:17.514091 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880c50b3d0bc44cead618b1eb7febdd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:17.519239 8839 tablet_bootstrap.cc:492] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Bootstrap starting.
I20260929 18:36:17.520951 8840 tablet_bootstrap.cc:492] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Bootstrap starting.
I20260929 18:36:17.521404 8839 tablet_bootstrap.cc:654] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:17.522228 8839 log.cc:824] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:17.522298 8841 tablet_bootstrap.cc:492] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Bootstrap starting.
I20260929 18:36:17.523576 8840 tablet_bootstrap.cc:654] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:17.524035 8839 tablet_bootstrap.cc:492] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: No bootstrap required, opened a new log
I20260929 18:36:17.524211 8839 ts_tablet_manager.cc:1397] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:36:17.524286 8841 tablet_bootstrap.cc:654] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:17.524362 8840 log.cc:824] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:17.525045 8841 log.cc:824] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:17.526307 8840 tablet_bootstrap.cc:492] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: No bootstrap required, opened a new log
I20260929 18:36:17.526491 8840 ts_tablet_manager.cc:1397] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:36:17.526583 8841 tablet_bootstrap.cc:492] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: No bootstrap required, opened a new log
I20260929 18:36:17.526759 8841 ts_tablet_manager.cc:1397] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:36:17.527957 8839 raft_consensus.cc:348] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.528218 8839 raft_consensus.cc:374] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:17.528272 8839 raft_consensus.cc:729] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90486608c3ef4ead8332f497801b318d, State: Initialized, Role: FOLLOWER
I20260929 18:36:17.528707 8839 consensus_queue.cc:260] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.529371 8811 heartbeater.cc:503] Master 127.0.198.62:38883 was elected leader, sending a full tablet report...
I20260929 18:36:17.529678 8839 ts_tablet_manager.cc:1428] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:36:17.529819 8841 raft_consensus.cc:348] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.530095 8841 raft_consensus.cc:374] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:17.530148 8841 raft_consensus.cc:729] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Initialized, Role: FOLLOWER
I20260929 18:36:17.530035 8840 raft_consensus.cc:348] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.530261 8840 raft_consensus.cc:374] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:17.530323 8840 raft_consensus.cc:729] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e4436f64d4143b1fb23fe26ce0e8d, State: Initialized, Role: FOLLOWER
I20260929 18:36:17.530731 8841 consensus_queue.cc:260] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.530728 8840 consensus_queue.cc:260] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.531620 8840 ts_tablet_manager.cc:1428] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:17.531632 8841 ts_tablet_manager.cc:1428] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:17.531937 8531 heartbeater.cc:503] Master 127.0.198.62:38883 was elected leader, sending a full tablet report...
I20260929 18:36:17.531949 8671 heartbeater.cc:503] Master 127.0.198.62:38883 was elected leader, sending a full tablet report...
W20260929 18:36:17.699653 8812 tablet.cc:2369] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:36:17.735288 8672 tablet.cc:2369] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:17.744215 8847 raft_consensus.cc:482] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:17.744407 8847 raft_consensus.cc:504] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.745519 8847 leader_election.cc:288] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059), 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235)
I20260929 18:36:17.749907 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.745886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:39671 (local address 127.0.198.1:39235)
0929 18:36:17.746046 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.746051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.746418 (+ 367us) server_negotiation.cc:407] Connection header received
0929 18:36:17.746600 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.746604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.746651 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.746713 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:17.747078 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.747724 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.748762 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.749320 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.749435 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.749504 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.749690 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.749776 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:36:17.750046 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.746012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:59153 (local address 127.0.198.3:41059)
0929 18:36:17.746139 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.746143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.746268 (+ 125us) server_negotiation.cc:407] Connection header received
0929 18:36:17.746406 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.746408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.746451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.746512 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:17.746775 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.747438 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.748949 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.749512 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.749606 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.749666 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.749835 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.749911 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:36:17.750052 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.745779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:41059 (local address 127.0.198.2:59153)
0929 18:36:17.746175 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.746187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.746300 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.746535 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.746538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.746554 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:17.746677 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.746682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.747560 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.747563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.748819 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.748828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.749621 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.749625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.749729 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.749751 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.749813 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.749889 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":105,"threads_started":1}
I20260929 18:36:17.750411 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.745786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:39235 (local address 127.0.198.2:39671)
0929 18:36:17.746320 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.746334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:17.746490 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:17.746832 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:17.746834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:17.746849 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:17.746946 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.746952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.747821 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.747825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.748637 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:17.748646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.749441 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.749445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:17.749537 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.749563 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:36:17.750073 (+ 510us) client_negotiation.cc:240] Negotiation successful
0929 18:36:17.750267 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":262,"thread_start_us":64,"threads_started":1}
I20260929 18:36:17.750744 8748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880c50b3d0bc44cead618b1eb7febdd6" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d" is_pre_election: true
I20260929 18:36:17.751029 8748 raft_consensus.cc:2454] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 0.
I20260929 18:36:17.751317 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880c50b3d0bc44cead618b1eb7febdd6" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" is_pre_election: true
I20260929 18:36:17.751513 8556 leader_election.cc:302] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90486608c3ef4ead8332f497801b318d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:17.751606 8485 raft_consensus.cc:2454] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 0.
I20260929 18:36:17.751797 8847 raft_consensus.cc:2781] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:17.751935 8847 raft_consensus.cc:482] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:17.751993 8847 raft_consensus.cc:3037] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:17.753060 8847 raft_consensus.cc:504] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.753403 8847 leader_election.cc:288] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 election: Requested vote from peers 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059), 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235)
I20260929 18:36:17.753674 8748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880c50b3d0bc44cead618b1eb7febdd6" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d"
I20260929 18:36:17.753753 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880c50b3d0bc44cead618b1eb7febdd6" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d"
I20260929 18:36:17.753823 8748 raft_consensus.cc:3037] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:17.753854 8485 raft_consensus.cc:3037] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:17.755116 8485 raft_consensus.cc:2454] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 1.
I20260929 18:36:17.755442 8559 leader_election.cc:302] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360e4436f64d4143b1fb23fe26ce0e8d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:17.755476 8748 raft_consensus.cc:2454] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 1.
I20260929 18:36:17.755719 8847 raft_consensus.cc:2781] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:17.755900 8847 raft_consensus.cc:686] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 LEADER]: Becoming Leader. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Running, Role: LEADER
I20260929 18:36:17.756245 8847 consensus_queue.cc:237] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } }
I20260929 18:36:17.759210 8327 catalog_manager.cc:5495] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } health_report { overall_health: HEALTHY } } }
W20260929 18:36:17.760347 8532 tablet.cc:2369] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:17.799861 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.796606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.198.2:44965)
0929 18:36:17.796718 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:17.796721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:17.796734 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:17.796820 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:17.796823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:17.796879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:17.796960 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:17.797250 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.797755 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.798603 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:17.798857 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:17.798951 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:17.799133 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:17.799581 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:17.799669 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:17.799697 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 18:36:17.799749 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:36:17.805548 8748 raft_consensus.cc:1261] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Refusing update from remote peer 91f661e2f82c4ae6aeda423adbf82684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:17.805549 8485 raft_consensus.cc:1261] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Refusing update from remote peer 91f661e2f82c4ae6aeda423adbf82684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:17.806249 8850 consensus_queue.cc:1035] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:17.806433 8847 consensus_queue.cc:1035] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:17.813365 8853 mvcc.cc:205] Tried to move back new op lower bound from 7334735781084930048 to 7334735780892127232. Current Snapshot: MvccSnapshot[applied={T|T < 7334735781084930048}]
I20260929 18:36:17.817849 8854 mvcc.cc:205] Tried to move back new op lower bound from 7334735781084930048 to 7334735780892127232. Current Snapshot: MvccSnapshot[applied={T|T < 7334735781084930048 or (T in {7334735781084930048})}]
I20260929 18:36:17.822721 8327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:17.822918 8327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:17.825438 8327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Sending DeleteTablet for 3 replicas of tablet 880c50b3d0bc44cead618b1eb7febdd6
I20260929 18:36:17.826293 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet 880c50b3d0bc44cead618b1eb7febdd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:17 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260929 18:36:17.826288 8745 tablet_service.cc:1514] Processing DeleteTablet for tablet 880c50b3d0bc44cead618b1eb7febdd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:17 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:57112
I20260929 18:36:17.826318 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet 880c50b3d0bc44cead618b1eb7febdd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:17 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:43390
I20260929 18:36:17.826978 8864 tablet_replica.cc:330] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: stopping tablet replica
I20260929 18:36:17.827291 8864 raft_consensus.cc:2229] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:17.827605 8864 raft_consensus.cc:2258] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:17.827704 8327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:36:17.827899 8862 tablet_replica.cc:330] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: stopping tablet replica
I20260929 18:36:17.828112 8862 raft_consensus.cc:2229] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 LEADER]: Raft consensus shutting down.
W20260929 18:36:17.828119 8327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:17.828397 8862 raft_consensus.cc:2258] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:17.828662 8863 tablet_replica.cc:330] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: stopping tablet replica
I20260929 18:36:17.828912 8863 raft_consensus.cc:2229] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:17.829190 8863 raft_consensus.cc:2258] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:17.830561 8864 ts_tablet_manager.cc:1905] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:17.830610 8862 ts_tablet_manager.cc:1905] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:17.830681 8863 ts_tablet_manager.cc:1905] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:17.833248 8863 ts_tablet_manager.cc:1918] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:17.833246 8864 ts_tablet_manager.cc:1918] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:17.833259 8862 ts_tablet_manager.cc:1918] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:17.833352 8862 log.cc:1196] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/880c50b3d0bc44cead618b1eb7febdd6
I20260929 18:36:17.833353 8863 log.cc:1196] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/880c50b3d0bc44cead618b1eb7febdd6
I20260929 18:36:17.833526 8864 log.cc:1196] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/880c50b3d0bc44cead618b1eb7febdd6
I20260929 18:36:17.833657 8863 ts_tablet_manager.cc:1939] T 880c50b3d0bc44cead618b1eb7febdd6 P 90486608c3ef4ead8332f497801b318d: Deleting consensus metadata
I20260929 18:36:17.833665 8862 ts_tablet_manager.cc:1939] T 880c50b3d0bc44cead618b1eb7febdd6 P 91f661e2f82c4ae6aeda423adbf82684: Deleting consensus metadata
I20260929 18:36:17.833900 8864 ts_tablet_manager.cc:1939] T 880c50b3d0bc44cead618b1eb7febdd6 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting consensus metadata
I20260929 18:36:17.833945 8745 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccd219bd62842ffa8267680099958a7 (DEFAULT_TABLE table=test-table [id=c8609f5b3cc24e4a8b507b9b8aae7288]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:17.834219 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccd219bd62842ffa8267680099958a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:17.834975 8464 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccd219bd62842ffa8267680099958a7 (DEFAULT_TABLE table=test-table [id=c8609f5b3cc24e4a8b507b9b8aae7288]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:17.835228 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccd219bd62842ffa8267680099958a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:17.836328 8839 tablet_bootstrap.cc:492] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: Bootstrap starting.
I20260929 18:36:17.836555 8315 catalog_manager.cc:4841] TS 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235): tablet 880c50b3d0bc44cead618b1eb7febdd6 (table test-table [id=759d451dd56248a0994bcd63b0393a9b]) successfully deleted
I20260929 18:36:17.837205 8840 tablet_bootstrap.cc:492] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: Bootstrap starting.
I20260929 18:36:17.837284 8839 tablet_bootstrap.cc:654] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:17.837611 8605 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccd219bd62842ffa8267680099958a7 (DEFAULT_TABLE table=test-table [id=c8609f5b3cc24e4a8b507b9b8aae7288]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:17.837846 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccd219bd62842ffa8267680099958a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:17.838116 8840 tablet_bootstrap.cc:654] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:17.838331 8314 catalog_manager.cc:4841] TS 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965): tablet 880c50b3d0bc44cead618b1eb7febdd6 (table test-table [id=759d451dd56248a0994bcd63b0393a9b]) successfully deleted
I20260929 18:36:17.839149 8312 catalog_manager.cc:4841] TS 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059): tablet 880c50b3d0bc44cead618b1eb7febdd6 (table test-table [id=759d451dd56248a0994bcd63b0393a9b]) successfully deleted
I20260929 18:36:17.840030 8839 tablet_bootstrap.cc:492] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: No bootstrap required, opened a new log
I20260929 18:36:17.840076 8840 tablet_bootstrap.cc:492] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: No bootstrap required, opened a new log
I20260929 18:36:17.840116 8839 ts_tablet_manager.cc:1397] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:36:17.840147 8840 ts_tablet_manager.cc:1397] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:17.840154 8841 tablet_bootstrap.cc:492] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: Bootstrap starting.
I20260929 18:36:17.840544 8839 raft_consensus.cc:348] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:17.840653 8839 raft_consensus.cc:374] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:17.840683 8839 raft_consensus.cc:729] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90486608c3ef4ead8332f497801b318d, State: Initialized, Role: FOLLOWER
I20260929 18:36:17.840735 8840 raft_consensus.cc:348] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:17.840842 8840 raft_consensus.cc:374] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:17.840816 8839 consensus_queue.cc:260] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:17.840874 8840 raft_consensus.cc:729] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e4436f64d4143b1fb23fe26ce0e8d, State: Initialized, Role: FOLLOWER
I20260929 18:36:17.840996 8840 consensus_queue.cc:260] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:17.841073 8839 ts_tablet_manager.cc:1428] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:17.841252 8840 ts_tablet_manager.cc:1428] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:17.841418 8841 tablet_bootstrap.cc:654] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:17.842635 8841 tablet_bootstrap.cc:492] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: No bootstrap required, opened a new log
I20260929 18:36:17.842715 8841 ts_tablet_manager.cc:1397] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:17.843153 8841 raft_consensus.cc:348] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:17.843268 8841 raft_consensus.cc:374] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:17.843300 8841 raft_consensus.cc:729] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Initialized, Role: FOLLOWER
I20260929 18:36:17.843416 8841 consensus_queue.cc:260] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:17.843659 8841 ts_tablet_manager.cc:1428] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:18.071025 8846 raft_consensus.cc:482] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:18.071168 8846 raft_consensus.cc:504] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.072243 8846 leader_election.cc:288] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965), 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059)
I20260929 18:36:18.077348 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.072512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44965 (local address 127.0.198.1:55355)
0929 18:36:18.072844 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.072858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.072997 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.073322 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.073325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.073356 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:18.073510 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.073517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.074629 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.074632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.076119 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.076132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.076947 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.076950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.077042 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.077065 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.077127 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.077190 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":122,"spinlock_wait_cycles":30848,"thread_start_us":73,"threads_started":1}
I20260929 18:36:18.077368 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.072588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:55355 (local address 127.0.198.2:44965)
0929 18:36:18.072716 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.072720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.072930 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:36:18.073123 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.073126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.073176 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.073259 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:18.073663 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.074477 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.076297 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.076816 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.076930 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.076999 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.077164 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.077242 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:18.077896 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.072561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:41059 (local address 127.0.198.1:60595)
0929 18:36:18.072853 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.072864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.073024 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.073322 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.073325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.073356 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:18.073517 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.073523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.074612 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.074616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.076119 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.076132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.077447 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.077450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.077546 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.077569 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.077671 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.077755 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":101,"spinlock_wait_cycles":18176,"thread_start_us":116,"threads_started":1}
I20260929 18:36:18.077893 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.072669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:60595 (local address 127.0.198.3:41059)
0929 18:36:18.072881 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.072884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.072957 (+ 73us) server_negotiation.cc:407] Connection header received
0929 18:36:18.073125 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.073128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.073176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.073259 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:18.073659 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.074477 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.076771 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.077310 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.077418 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.077476 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.077713 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.077792 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 18:36:18.077886 8625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccd219bd62842ffa8267680099958a7" candidate_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f661e2f82c4ae6aeda423adbf82684" is_pre_election: true
I20260929 18:36:18.078164 8625 raft_consensus.cc:2454] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360e4436f64d4143b1fb23fe26ce0e8d in term 0.
I20260929 18:36:18.078473 8765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccd219bd62842ffa8267680099958a7" candidate_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d" is_pre_election: true
I20260929 18:36:18.078613 8765 raft_consensus.cc:2454] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360e4436f64d4143b1fb23fe26ce0e8d in term 0.
I20260929 18:36:18.078591 8418 leader_election.cc:302] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360e4436f64d4143b1fb23fe26ce0e8d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:18.078876 8846 raft_consensus.cc:2781] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:18.078948 8846 raft_consensus.cc:482] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:18.078985 8846 raft_consensus.cc:3037] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.080129 8846 raft_consensus.cc:504] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.080505 8846 leader_election.cc:288] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [CANDIDATE]: Term 1 election: Requested vote from peers 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965), 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059)
I20260929 18:36:18.080858 8765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccd219bd62842ffa8267680099958a7" candidate_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d"
I20260929 18:36:18.081001 8765 raft_consensus.cc:3037] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.081005 8625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccd219bd62842ffa8267680099958a7" candidate_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f661e2f82c4ae6aeda423adbf82684"
I20260929 18:36:18.081120 8625 raft_consensus.cc:3037] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.082212 8625 raft_consensus.cc:2454] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360e4436f64d4143b1fb23fe26ce0e8d in term 1.
I20260929 18:36:18.082221 8765 raft_consensus.cc:2454] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360e4436f64d4143b1fb23fe26ce0e8d in term 1.
I20260929 18:36:18.082531 8416 leader_election.cc:302] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360e4436f64d4143b1fb23fe26ce0e8d, 90486608c3ef4ead8332f497801b318d; no voters:
I20260929 18:36:18.082757 8846 raft_consensus.cc:2781] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:18.082985 8846 raft_consensus.cc:686] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 LEADER]: Becoming Leader. State: Replica: 360e4436f64d4143b1fb23fe26ce0e8d, State: Running, Role: LEADER
I20260929 18:36:18.083385 8846 consensus_queue.cc:237] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.085755 8327 catalog_manager.cc:5495] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1). New cstate: current_term: 1 leader_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:36:18 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.62@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:18.164176 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.155693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38883 (local address 127.0.0.1:46476)
0929 18:36:18.156107 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.156120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.156237 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.156486 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.156488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.156919 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:18.157010 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.157017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.157766 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.157770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.158379 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.158388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.158785 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.158788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.158858 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.159446 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:18.159459 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:18.160880 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:18.162508 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:18.162516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:18.162522 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:18.162805 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:18.163060 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:18.163062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:18.163063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:18.163147 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:18.163561 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:18.163567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:18.163680 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.163841 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.163982 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":129,"threads_started":1}
I20260929 18:36:18.164184 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.155775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.0.198.62:38883)
0929 18:36:18.156056 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.156065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.156194 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:36:18.156346 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.156348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.156387 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.156459 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:18.157127 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.157664 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.158483 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.158691 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.158780 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.161013 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:18.161017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:18.161018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:18.161040 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:18.162369 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:18.162903 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:18.162906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:18.162907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:18.162986 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:18.163232 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:18.163234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:18.163235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:18.163430 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:18.163505 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.163903 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.164042 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260929 18:36:18.168618 8327 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:46476:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:36:18.169368 8327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:18.174077 8464 tablet_service.cc:1467] Processing CreateTablet for tablet 982b86a4f7d94dee88f8d2973bdb8bde (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4200aff0fa2b4f5a83fdcde587307a55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:36:18.174196 8745 tablet_service.cc:1467] Processing CreateTablet for tablet 982b86a4f7d94dee88f8d2973bdb8bde (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4200aff0fa2b4f5a83fdcde587307a55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:36:18.174276 8605 tablet_service.cc:1467] Processing CreateTablet for tablet 982b86a4f7d94dee88f8d2973bdb8bde (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4200aff0fa2b4f5a83fdcde587307a55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:36:18.174495 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982b86a4f7d94dee88f8d2973bdb8bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.174562 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982b86a4f7d94dee88f8d2973bdb8bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.174594 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982b86a4f7d94dee88f8d2973bdb8bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.176816 8840 tablet_bootstrap.cc:492] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: Bootstrap starting.
I20260929 18:36:18.177145 8841 tablet_bootstrap.cc:492] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: Bootstrap starting.
I20260929 18:36:18.177209 8839 tablet_bootstrap.cc:492] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: Bootstrap starting.
I20260929 18:36:18.177743 8840 tablet_bootstrap.cc:654] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.177879 8841 tablet_bootstrap.cc:654] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.178143 8839 tablet_bootstrap.cc:654] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.178928 8841 tablet_bootstrap.cc:492] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: No bootstrap required, opened a new log
I20260929 18:36:18.179011 8841 ts_tablet_manager.cc:1397] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:18.179127 8840 tablet_bootstrap.cc:492] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: No bootstrap required, opened a new log
I20260929 18:36:18.179214 8840 ts_tablet_manager.cc:1397] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:18.179221 8839 tablet_bootstrap.cc:492] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: No bootstrap required, opened a new log
I20260929 18:36:18.179287 8839 ts_tablet_manager.cc:1397] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:18.179409 8841 raft_consensus.cc:348] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.179541 8841 raft_consensus.cc:374] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.179582 8841 raft_consensus.cc:729] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.179659 8839 raft_consensus.cc:348] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.179644 8840 raft_consensus.cc:348] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.179703 8841 consensus_queue.cc:260] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.179757 8840 raft_consensus.cc:374] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.179757 8839 raft_consensus.cc:374] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.179790 8840 raft_consensus.cc:729] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e4436f64d4143b1fb23fe26ce0e8d, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.179806 8839 raft_consensus.cc:729] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90486608c3ef4ead8332f497801b318d, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.179900 8840 consensus_queue.cc:260] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.179947 8841 ts_tablet_manager.cc:1428] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:18.179922 8839 consensus_queue.cc:260] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.180147 8840 ts_tablet_manager.cc:1428] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:18.180167 8839 ts_tablet_manager.cc:1428] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:18.180122 8850 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.180274 8850 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:36:18.180431 8845 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.180559 8845 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:36:18.181010 8846 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.181129 8846 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:36:18.374594 8850 raft_consensus.cc:482] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:18.374754 8850 raft_consensus.cc:504] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.375186 8850 leader_election.cc:288] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235), 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059)
I20260929 18:36:18.375667 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982b86a4f7d94dee88f8d2973bdb8bde" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" is_pre_election: true
I20260929 18:36:18.375674 8765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982b86a4f7d94dee88f8d2973bdb8bde" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d" is_pre_election: true
I20260929 18:36:18.375813 8485 raft_consensus.cc:2454] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 0.
I20260929 18:36:18.375825 8765 raft_consensus.cc:2454] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 0.
I20260929 18:36:18.376151 8556 leader_election.cc:302] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 360e4436f64d4143b1fb23fe26ce0e8d, 90486608c3ef4ead8332f497801b318d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:18.376322 8850 raft_consensus.cc:2781] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:18.376381 8850 raft_consensus.cc:482] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:18.376408 8850 raft_consensus.cc:3037] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.377398 8850 raft_consensus.cc:504] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.377743 8850 leader_election.cc:288] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 election: Requested vote from peers 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235), 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059)
I20260929 18:36:18.378082 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982b86a4f7d94dee88f8d2973bdb8bde" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d"
I20260929 18:36:18.378214 8485 raft_consensus.cc:3037] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.378188 8765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982b86a4f7d94dee88f8d2973bdb8bde" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d"
I20260929 18:36:18.378329 8765 raft_consensus.cc:3037] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.379328 8485 raft_consensus.cc:2454] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 1.
I20260929 18:36:18.379328 8765 raft_consensus.cc:2454] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 1.
I20260929 18:36:18.379628 8559 leader_election.cc:302] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360e4436f64d4143b1fb23fe26ce0e8d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:18.379801 8850 raft_consensus.cc:2781] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:18.379971 8850 raft_consensus.cc:686] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [term 1 LEADER]: Becoming Leader. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Running, Role: LEADER
I20260929 18:36:18.380115 8850 consensus_queue.cc:237] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.381173 8882 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: TxnStatusTablet state changed. Reason: New leader 91f661e2f82c4ae6aeda423adbf82684. Latest consensus state: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.381296 8882 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:36:18.381512 8884 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:36:18.381656 8884 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:36:18.382002 8328 catalog_manager.cc:5495] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20260929 18:36:18.420401 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.417142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44965 (local address 127.0.0.1:43404)
0929 18:36:18.417320 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.417337 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.417456 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.417720 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.417722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.417732 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:36:18.417831 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.417836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.418605 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.418609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.419225 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.419234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.419669 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.419673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.419746 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.419814 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.420082 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.420097 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.420156 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.420250 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:36:18.420387 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.417270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.0.198.2:44965)
0929 18:36:18.417399 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.417403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.417417 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:18.417573 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.417576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.417626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.417690 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:18.417925 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.418503 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.419341 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.419555 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.419642 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.419933 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.420010 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.420067 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.420173 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.420225 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:36:18.423404 8765 raft_consensus.cc:1261] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Refusing update from remote peer 91f661e2f82c4ae6aeda423adbf82684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.423594 8485 raft_consensus.cc:1261] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Refusing update from remote peer 91f661e2f82c4ae6aeda423adbf82684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.423976 8882 consensus_queue.cc:1035] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.424124 8850 consensus_queue.cc:1035] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.425387 8846 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: TxnStatusTablet state changed. Reason: New leader 91f661e2f82c4ae6aeda423adbf82684. Latest consensus state: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.425504 8846 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:36:18.425442 8845 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: TxnStatusTablet state changed. Reason: New leader 91f661e2f82c4ae6aeda423adbf82684. Latest consensus state: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.425550 8845 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:36:18.427481 8850 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.427598 8850 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:36:18.428151 8846 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.428272 8846 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 360e4436f64d4143b1fb23fe26ce0e8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:36:18.431718 8850 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.431844 8850 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 91f661e2f82c4ae6aeda423adbf82684: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:36:18.433012 8845 tablet_replica.cc:439] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } } }
I20260929 18:36:18.433125 8845 tablet_replica.cc:442] T 982b86a4f7d94dee88f8d2973bdb8bde P 90486608c3ef4ead8332f497801b318d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:36:18.433784 8889 mvcc.cc:205] Tried to move back new op lower bound from 7334735783618236416 to 7334735783446552576. Current Snapshot: MvccSnapshot[applied={T|T < 7334735783618236416 or (T in {7334735783618236416})}]
I20260929 18:36:18.438758 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.434640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.198.1:39235)
0929 18:36:18.434838 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.434842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.434854 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:18.434999 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.435001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.435055 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.435192 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:18.435440 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.435949 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.437209 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.437496 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.437583 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.437861 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.438380 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.438467 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.438548 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.438606 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260929 18:36:18.449806 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.445820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:41903 (local address 127.0.198.2:44965)
0929 18:36:18.445961 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.445989 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.446072 (+ 83us) server_negotiation.cc:407] Connection header received
0929 18:36:18.446169 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.446172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.446224 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.446301 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:18.446616 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.447383 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.448575 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.448918 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.449018 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.449362 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.449454 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.449515 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.449610 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.449666 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:36:18.449834 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.445698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44965 (local address 127.0.198.1:41903)
0929 18:36:18.445987 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.446002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.446096 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.446391 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.446394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.446401 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:36:18.446510 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.446515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.447528 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.447532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.448440 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.448452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.449047 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.449050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.449148 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.449204 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.449531 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.449537 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.449593 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.449677 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260929 18:36:18.465492 8765 raft_consensus.cc:1261] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Refusing update from remote peer 360e4436f64d4143b1fb23fe26ce0e8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.465658 8625 raft_consensus.cc:1261] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Refusing update from remote peer 360e4436f64d4143b1fb23fe26ce0e8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.465989 8870 consensus_queue.cc:1035] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.466229 8846 consensus_queue.cc:1035] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.489571 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.485208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:39235 (local address 127.0.198.2:50237)
0929 18:36:18.485622 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.485634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.485726 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.486023 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.486026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.486035 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:36:18.486149 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.486155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.487196 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.487200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.488113 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.488136 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.488796 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.488799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.488888 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.488951 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.489288 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.489295 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.489362 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.489446 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":109,"threads_started":1}
I20260929 18:36:18.490079 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.485311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:50237 (local address 127.0.198.1:39235)
0929 18:36:18.485442 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.485447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.485702 (+ 255us) server_negotiation.cc:407] Connection header received
0929 18:36:18.485841 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.485844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.485896 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.485993 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:18.486260 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.486968 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.488244 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.488473 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.488572 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.489109 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.489183 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.489248 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.489871 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.489928 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:18.517318 8326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:18.517498 8326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:18.519840 8326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Sending DeleteTablet for 3 replicas of tablet 5ccd219bd62842ffa8267680099958a7
I20260929 18:36:18.520440 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccd219bd62842ffa8267680099958a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260929 18:36:18.520573 8745 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccd219bd62842ffa8267680099958a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:57112
I20260929 18:36:18.521003 8909 tablet_replica.cc:330] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: stopping tablet replica
I20260929 18:36:18.521174 8909 raft_consensus.cc:2229] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:18.521229 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccd219bd62842ffa8267680099958a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:43390
I20260929 18:36:18.521623 8909 raft_consensus.cc:2258] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.522523 8909 ts_tablet_manager.cc:1905] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.525600 8909 ts_tablet_manager.cc:1918] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:36:18.525745 8909 log.cc:1196] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/5ccd219bd62842ffa8267680099958a7
I20260929 18:36:18.526093 8909 ts_tablet_manager.cc:1939] T 5ccd219bd62842ffa8267680099958a7 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting consensus metadata
I20260929 18:36:18.526481 8910 tablet_replica.cc:330] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: stopping tablet replica
I20260929 18:36:18.526636 8910 raft_consensus.cc:2229] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:18.526757 8910 raft_consensus.cc:2258] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.527420 8910 ts_tablet_manager.cc:1905] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.527920 8315 catalog_manager.cc:4841] TS 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235): tablet 5ccd219bd62842ffa8267680099958a7 (table test-table [id=c8609f5b3cc24e4a8b507b9b8aae7288]) successfully deleted
I20260929 18:36:18.529753 8911 tablet_replica.cc:330] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: stopping tablet replica
I20260929 18:36:18.529906 8911 raft_consensus.cc:2229] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:18.530058 8911 raft_consensus.cc:2258] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.530216 8910 ts_tablet_manager.cc:1918] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:36:18.530305 8910 log.cc:1196] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/5ccd219bd62842ffa8267680099958a7
I20260929 18:36:18.530584 8910 ts_tablet_manager.cc:1939] T 5ccd219bd62842ffa8267680099958a7 P 90486608c3ef4ead8332f497801b318d: Deleting consensus metadata
I20260929 18:36:18.530767 8911 ts_tablet_manager.cc:1905] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.531307 8312 catalog_manager.cc:4841] TS 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059): tablet 5ccd219bd62842ffa8267680099958a7 (table test-table [id=c8609f5b3cc24e4a8b507b9b8aae7288]) successfully deleted
I20260929 18:36:18.533255 8911 ts_tablet_manager.cc:1918] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:36:18.533324 8911 log.cc:1196] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/5ccd219bd62842ffa8267680099958a7
I20260929 18:36:18.533527 8911 ts_tablet_manager.cc:1939] T 5ccd219bd62842ffa8267680099958a7 P 91f661e2f82c4ae6aeda423adbf82684: Deleting consensus metadata
I20260929 18:36:18.534420 8314 catalog_manager.cc:4841] TS 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965): tablet 5ccd219bd62842ffa8267680099958a7 (table test-table [id=c8609f5b3cc24e4a8b507b9b8aae7288]) successfully deleted
Sep 29 18:36:18 dist-test-slave-q5qx krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:36:18 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:18 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:36:18.550479 8878 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:18.543856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.0.198.62:38883)
0929 18:36:18.543997 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.544002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.544132 (+ 130us) server_negotiation.cc:407] Connection header received
0929 18:36:18.544256 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.544260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.544323 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.544411 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:18.545107 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.545930 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.546830 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.547195 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.547305 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.550347 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46488: BlockingRecv error: recv got EOF from 127.0.0.1:46488 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 18:36:18 dist-test-slave-q5qx krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM
I20260929 18:36:18.560307 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.551878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.0.198.62:38883)
0929 18:36:18.551990 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.551994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.552098 (+ 104us) server_negotiation.cc:407] Connection header received
0929 18:36:18.552212 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.552214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.552253 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.552316 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:18.552979 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.553519 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.554652 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.554901 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.554979 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.557479 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:18.557482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:18.557484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:18.557509 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:18.558831 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:18.559252 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:18.559255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:18.559256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:18.559324 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:18.559549 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:18.559551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:18.559552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:18.559720 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:18.559838 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.560030 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.560124 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:36:18.562856 8326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:18.563266 8326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:18.567919 8605 tablet_service.cc:1467] Processing CreateTablet for tablet bc4d49c487ed4b2fb5eeb26ebcf82520 (DEFAULT_TABLE table=test-table [id=42db257b486d4e72ab69db59ee1fef5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:18.568059 8464 tablet_service.cc:1467] Processing CreateTablet for tablet bc4d49c487ed4b2fb5eeb26ebcf82520 (DEFAULT_TABLE table=test-table [id=42db257b486d4e72ab69db59ee1fef5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:18.568230 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4d49c487ed4b2fb5eeb26ebcf82520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.568323 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4d49c487ed4b2fb5eeb26ebcf82520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.570629 8840 tablet_bootstrap.cc:492] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: Bootstrap starting.
I20260929 18:36:18.570845 8745 tablet_service.cc:1467] Processing CreateTablet for tablet bc4d49c487ed4b2fb5eeb26ebcf82520 (DEFAULT_TABLE table=test-table [id=42db257b486d4e72ab69db59ee1fef5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:18.571105 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4d49c487ed4b2fb5eeb26ebcf82520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.571427 8840 tablet_bootstrap.cc:654] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.572671 8840 tablet_bootstrap.cc:492] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: No bootstrap required, opened a new log
I20260929 18:36:18.572748 8840 ts_tablet_manager.cc:1397] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:18.573151 8840 raft_consensus.cc:348] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.573264 8840 raft_consensus.cc:374] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.573266 8839 tablet_bootstrap.cc:492] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: Bootstrap starting.
I20260929 18:36:18.573297 8840 raft_consensus.cc:729] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e4436f64d4143b1fb23fe26ce0e8d, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.573352 8841 tablet_bootstrap.cc:492] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: Bootstrap starting.
I20260929 18:36:18.573412 8840 consensus_queue.cc:260] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.573655 8840 ts_tablet_manager.cc:1428] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:18.574162 8841 tablet_bootstrap.cc:654] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.574169 8839 tablet_bootstrap.cc:654] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.575523 8841 tablet_bootstrap.cc:492] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: No bootstrap required, opened a new log
I20260929 18:36:18.575603 8841 ts_tablet_manager.cc:1397] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:18.575976 8841 raft_consensus.cc:348] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.576076 8839 tablet_bootstrap.cc:492] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: No bootstrap required, opened a new log
I20260929 18:36:18.576076 8841 raft_consensus.cc:374] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.576104 8841 raft_consensus.cc:729] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.576135 8839 ts_tablet_manager.cc:1397] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 18:36:18.576224 8841 consensus_queue.cc:260] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.576474 8841 ts_tablet_manager.cc:1428] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:36:18.576525 8839 raft_consensus.cc:348] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.576644 8839 raft_consensus.cc:374] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.576684 8839 raft_consensus.cc:729] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90486608c3ef4ead8332f497801b318d, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.576795 8839 consensus_queue.cc:260] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.577837 8839 ts_tablet_manager.cc:1428] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:36:18.712525 8845 raft_consensus.cc:482] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:18.712683 8845 raft_consensus.cc:504] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.713804 8845 leader_election.cc:288] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235), 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965)
I20260929 18:36:18.717515 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.714161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:41011 (local address 127.0.198.2:44965)
0929 18:36:18.714289 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.714293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.714464 (+ 171us) server_negotiation.cc:407] Connection header received
0929 18:36:18.714596 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.714598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.714640 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.714702 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:18.715024 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.715553 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.716594 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.716981 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.717064 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.717136 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.717309 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.717379 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:36:18.717523 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.714162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:44625 (local address 127.0.198.1:39235)
0929 18:36:18.714352 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.714357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.714436 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:36:18.714576 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.714579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.714621 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.714676 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:18.715006 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.715511 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.716600 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.716999 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.717092 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.717152 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.717341 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.717409 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 18:36:18.717557 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.714059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44965 (local address 127.0.198.3:41011)
0929 18:36:18.714373 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.714386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.714488 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.714724 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.714726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.714803 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:18.714917 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.714922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.715662 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.715664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.716481 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.716489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.717097 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.717100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.717194 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.717218 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.717276 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.717383 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":42,"spinlock_wait_cycles":896,"thread_start_us":68,"threads_started":1}
I20260929 18:36:18.717564 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.714059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:39235 (local address 127.0.198.3:44625)
0929 18:36:18.714345 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.714357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.714468 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.714703 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.714705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.714720 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:18.714894 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.714900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.715612 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.715616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.716480 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.716489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.717106 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.717110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.717225 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.717242 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.717312 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.717385 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":26,"spinlock_wait_cycles":127872,"thread_start_us":78,"threads_started":1}
I20260929 18:36:18.718042 8625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4d49c487ed4b2fb5eeb26ebcf82520" candidate_uuid: "90486608c3ef4ead8332f497801b318d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f661e2f82c4ae6aeda423adbf82684" is_pre_election: true
I20260929 18:36:18.718154 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4d49c487ed4b2fb5eeb26ebcf82520" candidate_uuid: "90486608c3ef4ead8332f497801b318d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" is_pre_election: true
I20260929 18:36:18.718196 8625 raft_consensus.cc:2454] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90486608c3ef4ead8332f497801b318d in term 0.
I20260929 18:36:18.718294 8485 raft_consensus.cc:2454] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90486608c3ef4ead8332f497801b318d in term 0.
I20260929 18:36:18.718516 8698 leader_election.cc:302] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90486608c3ef4ead8332f497801b318d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:18.718748 8845 raft_consensus.cc:2781] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:18.718803 8845 raft_consensus.cc:482] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:18.718850 8845 raft_consensus.cc:3037] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.719835 8845 raft_consensus.cc:504] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.720197 8845 leader_election.cc:288] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [CANDIDATE]: Term 1 election: Requested vote from peers 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235), 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965)
I20260929 18:36:18.720551 8625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4d49c487ed4b2fb5eeb26ebcf82520" candidate_uuid: "90486608c3ef4ead8332f497801b318d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f661e2f82c4ae6aeda423adbf82684"
I20260929 18:36:18.720571 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4d49c487ed4b2fb5eeb26ebcf82520" candidate_uuid: "90486608c3ef4ead8332f497801b318d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d"
I20260929 18:36:18.720678 8625 raft_consensus.cc:3037] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.720680 8485 raft_consensus.cc:3037] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.721698 8625 raft_consensus.cc:2454] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90486608c3ef4ead8332f497801b318d in term 1.
I20260929 18:36:18.721706 8485 raft_consensus.cc:2454] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90486608c3ef4ead8332f497801b318d in term 1.
I20260929 18:36:18.721998 8699 leader_election.cc:302] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360e4436f64d4143b1fb23fe26ce0e8d, 90486608c3ef4ead8332f497801b318d; no voters:
I20260929 18:36:18.722164 8845 raft_consensus.cc:2781] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:18.722368 8845 raft_consensus.cc:686] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 1 LEADER]: Becoming Leader. State: Replica: 90486608c3ef4ead8332f497801b318d, State: Running, Role: LEADER
I20260929 18:36:18.722632 8845 consensus_queue.cc:237] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.724776 8328 catalog_manager.cc:5495] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90486608c3ef4ead8332f497801b318d (127.0.198.3). New cstate: current_term: 1 leader_uuid: "90486608c3ef4ead8332f497801b318d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } health_report { overall_health: HEALTHY } } }
I20260929 18:36:18.769137 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.765281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.0.198.3:41059)
0929 18:36:18.765535 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.765541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.765553 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:18.765629 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.765631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.765686 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.765803 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:18.766080 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.766628 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.767656 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.767899 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.767976 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.768327 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.768787 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.768871 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.768960 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.769007 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260929 18:36:18.771749 8485 raft_consensus.cc:1261] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Refusing update from remote peer 90486608c3ef4ead8332f497801b318d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.771821 8625 raft_consensus.cc:1261] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Refusing update from remote peer 90486608c3ef4ead8332f497801b318d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.772303 8845 consensus_queue.cc:1035] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [LEADER]: Connected to new peer: Peer: permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.772495 8929 consensus_queue.cc:1035] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.784993 8326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:18.785149 8326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:18.787307 8326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Sending DeleteTablet for 3 replicas of tablet bc4d49c487ed4b2fb5eeb26ebcf82520
I20260929 18:36:18.787969 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4d49c487ed4b2fb5eeb26ebcf82520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260929 18:36:18.788031 8745 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4d49c487ed4b2fb5eeb26ebcf82520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:57112
I20260929 18:36:18.788174 8909 tablet_replica.cc:330] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: stopping tablet replica
I20260929 18:36:18.788136 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4d49c487ed4b2fb5eeb26ebcf82520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:43390
I20260929 18:36:18.788209 8910 tablet_replica.cc:330] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: stopping tablet replica
I20260929 18:36:18.788278 8909 raft_consensus.cc:2229] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:18.788306 8910 raft_consensus.cc:2229] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:18.788327 8911 tablet_replica.cc:330] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: stopping tablet replica
I20260929 18:36:18.788416 8909 raft_consensus.cc:2258] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.788436 8911 raft_consensus.cc:2229] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:18.788559 8911 raft_consensus.cc:2258] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.788614 8910 raft_consensus.cc:2258] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.789211 8909 ts_tablet_manager.cc:1905] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.789318 8911 ts_tablet_manager.cc:1905] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.789279 8326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:18.789687 8326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:18.791524 8909 ts_tablet_manager.cc:1918] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:18.791601 8909 log.cc:1196] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/bc4d49c487ed4b2fb5eeb26ebcf82520
I20260929 18:36:18.791671 8911 ts_tablet_manager.cc:1918] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:18.791750 8911 log.cc:1196] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/bc4d49c487ed4b2fb5eeb26ebcf82520
I20260929 18:36:18.791841 8909 ts_tablet_manager.cc:1939] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 360e4436f64d4143b1fb23fe26ce0e8d: Deleting consensus metadata
I20260929 18:36:18.791993 8911 ts_tablet_manager.cc:1939] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 91f661e2f82c4ae6aeda423adbf82684: Deleting consensus metadata
I20260929 18:36:18.792773 8315 catalog_manager.cc:4841] TS 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235): tablet bc4d49c487ed4b2fb5eeb26ebcf82520 (table test-table [id=42db257b486d4e72ab69db59ee1fef5b]) successfully deleted
I20260929 18:36:18.793051 8910 ts_tablet_manager.cc:1905] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.794435 8464 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac (DEFAULT_TABLE table=test-table [id=d16ae230dcd443158f7172f712238a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:18.794709 8464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.794683 8745 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac (DEFAULT_TABLE table=test-table [id=d16ae230dcd443158f7172f712238a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:18.794893 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.795352 8910 ts_tablet_manager.cc:1918] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:18.795409 8910 log.cc:1196] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/bc4d49c487ed4b2fb5eeb26ebcf82520
I20260929 18:36:18.795622 8910 ts_tablet_manager.cc:1939] T bc4d49c487ed4b2fb5eeb26ebcf82520 P 90486608c3ef4ead8332f497801b318d: Deleting consensus metadata
I20260929 18:36:18.796198 8314 catalog_manager.cc:4841] TS 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965): tablet bc4d49c487ed4b2fb5eeb26ebcf82520 (table test-table [id=42db257b486d4e72ab69db59ee1fef5b]) successfully deleted
I20260929 18:36:18.796657 8605 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac (DEFAULT_TABLE table=test-table [id=d16ae230dcd443158f7172f712238a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:18.796900 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:18.796972 8839 tablet_bootstrap.cc:492] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: Bootstrap starting.
I20260929 18:36:18.797870 8839 tablet_bootstrap.cc:654] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.799113 8841 tablet_bootstrap.cc:492] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: Bootstrap starting.
I20260929 18:36:18.799165 8839 tablet_bootstrap.cc:492] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: No bootstrap required, opened a new log
I20260929 18:36:18.799183 8840 tablet_bootstrap.cc:492] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d: Bootstrap starting.
I20260929 18:36:18.799237 8839 ts_tablet_manager.cc:1397] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:36:18.799666 8839 raft_consensus.cc:348] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.799770 8839 raft_consensus.cc:374] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.799801 8839 raft_consensus.cc:729] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90486608c3ef4ead8332f497801b318d, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.799949 8841 tablet_bootstrap.cc:654] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.799921 8839 consensus_queue.cc:260] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.799993 8840 tablet_bootstrap.cc:654] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:18.800170 8839 ts_tablet_manager.cc:1428] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:18.801036 8840 tablet_bootstrap.cc:492] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d: No bootstrap required, opened a new log
I20260929 18:36:18.801116 8840 ts_tablet_manager.cc:1397] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:18.801496 8840 raft_consensus.cc:348] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.801605 8840 raft_consensus.cc:374] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.801635 8840 raft_consensus.cc:729] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e4436f64d4143b1fb23fe26ce0e8d, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.801748 8840 consensus_queue.cc:260] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.802024 8840 ts_tablet_manager.cc:1428] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:36:18.802865 8312 catalog_manager.cc:4841] TS 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059): tablet bc4d49c487ed4b2fb5eeb26ebcf82520 (table test-table [id=42db257b486d4e72ab69db59ee1fef5b]) successfully deleted
I20260929 18:36:18.803560 8841 tablet_bootstrap.cc:492] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: No bootstrap required, opened a new log
I20260929 18:36:18.803736 8841 ts_tablet_manager.cc:1397] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 18:36:18.804119 8841 raft_consensus.cc:348] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.804222 8841 raft_consensus.cc:374] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:18.804248 8841 raft_consensus.cc:729] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Initialized, Role: FOLLOWER
I20260929 18:36:18.804364 8841 consensus_queue.cc:260] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.804593 8841 ts_tablet_manager.cc:1428] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:36:18.884750 8850 raft_consensus.cc:482] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:18.884879 8850 raft_consensus.cc:504] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.885281 8850 leader_election.cc:288] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235), 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059)
I20260929 18:36:18.885751 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0c12a4fed84a5e8b20ad527e07f5ac" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" is_pre_election: true
I20260929 18:36:18.885767 8765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0c12a4fed84a5e8b20ad527e07f5ac" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d" is_pre_election: true
I20260929 18:36:18.885900 8485 raft_consensus.cc:2454] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 0.
I20260929 18:36:18.885931 8765 raft_consensus.cc:2454] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 0.
I20260929 18:36:18.886282 8559 leader_election.cc:302] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 360e4436f64d4143b1fb23fe26ce0e8d, 90486608c3ef4ead8332f497801b318d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:18.886483 8850 raft_consensus.cc:2781] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:18.886544 8850 raft_consensus.cc:482] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:18.886608 8850 raft_consensus.cc:3037] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.887557 8850 raft_consensus.cc:504] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.887920 8850 leader_election.cc:288] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 election: Requested vote from peers 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235), 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059)
I20260929 18:36:18.888252 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0c12a4fed84a5e8b20ad527e07f5ac" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e4436f64d4143b1fb23fe26ce0e8d"
I20260929 18:36:18.888293 8765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0c12a4fed84a5e8b20ad527e07f5ac" candidate_uuid: "91f661e2f82c4ae6aeda423adbf82684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486608c3ef4ead8332f497801b318d"
I20260929 18:36:18.888394 8485 raft_consensus.cc:3037] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.888398 8765 raft_consensus.cc:3037] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:18.889364 8765 raft_consensus.cc:2454] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 1.
I20260929 18:36:18.889518 8485 raft_consensus.cc:2454] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f661e2f82c4ae6aeda423adbf82684 in term 1.
I20260929 18:36:18.889678 8556 leader_election.cc:302] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90486608c3ef4ead8332f497801b318d, 91f661e2f82c4ae6aeda423adbf82684; no voters:
I20260929 18:36:18.889847 8850 raft_consensus.cc:2781] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:18.889926 8850 raft_consensus.cc:686] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 1 LEADER]: Becoming Leader. State: Replica: 91f661e2f82c4ae6aeda423adbf82684, State: Running, Role: LEADER
I20260929 18:36:18.890091 8850 consensus_queue.cc:237] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } }
I20260929 18:36:18.892031 8328 catalog_manager.cc:5495] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "91f661e2f82c4ae6aeda423adbf82684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f661e2f82c4ae6aeda423adbf82684" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 44965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20260929 18:36:18.914736 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.911189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.0.198.2:44965)
0929 18:36:18.911504 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.911509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.911521 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:18.911561 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.911564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.911626 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.911704 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:18.911941 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.912683 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.913546 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.913892 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.914033 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.914277 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.914356 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.914408 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.914553 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.914606 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260929 18:36:18.924167 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.920712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:41921 (local address 127.0.198.2:44965)
0929 18:36:18.920865 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:18.920868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:18.920880 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:18.921022 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:18.921024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:18.921064 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:18.921125 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:18.921491 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.922055 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.923211 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.923422 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.923502 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.923758 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.923833 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.923898 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:18.923969 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:36:18.924029 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":94}
I20260929 18:36:18.924211 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.920611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:44965 (local address 127.0.198.2:41921)
0929 18:36:18.920824 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.920840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:18.920926 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:18.921174 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:18.921177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:18.921185 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:36:18.921287 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.921294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.922203 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.922206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.923077 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:18.923088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:18.923518 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:18.923521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:18.923603 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:18.923633 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:36:18.923907 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:36:18.923912 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:18.923964 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:36:18.924079 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":19}
I20260929 18:36:18.930168 8485 raft_consensus.cc:1261] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Refusing update from remote peer 91f661e2f82c4ae6aeda423adbf82684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.930226 8765 raft_consensus.cc:1261] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Refusing update from remote peer 91f661e2f82c4ae6aeda423adbf82684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:18.930529 8894 consensus_queue.cc:1035] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360e4436f64d4143b1fb23fe26ce0e8d" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.931228 8894 consensus_queue.cc:1035] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90486608c3ef4ead8332f497801b318d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:18.965168 8327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:18.965338 8327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:18.967293 8327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac4cceb12ca4ba5841ea4cb85bf8c3e: Sending DeleteTablet for 3 replicas of tablet 2a0c12a4fed84a5e8b20ad527e07f5ac
I20260929 18:36:18.967904 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:43390
I20260929 18:36:18.968091 8911 tablet_replica.cc:330] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: stopping tablet replica
I20260929 18:36:18.968065 8464 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260929 18:36:18.968065 8745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:18 UTC) from {username='oryx', principal='oryx/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:57112
I20260929 18:36:18.968209 8911 raft_consensus.cc:2229] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:18.968225 8909 tablet_replica.cc:330] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d: stopping tablet replica
I20260929 18:36:18.968323 8909 raft_consensus.cc:2229] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:18.968410 8911 raft_consensus.cc:2258] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.968422 8909 raft_consensus.cc:2258] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 360e4436f64d4143b1fb23fe26ce0e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.968509 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 8400
I20260929 18:36:18.968582 8910 tablet_replica.cc:330] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: stopping tablet replica
I20260929 18:36:18.968693 8910 raft_consensus.cc:2229] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:18.968830 8910 raft_consensus.cc:2258] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:18.969686 8911 ts_tablet_manager.cc:1905] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.970721 8910 ts_tablet_manager.cc:1905] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:18.972355 8911 ts_tablet_manager.cc:1918] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:36:18.972434 8911 log.cc:1196] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/2a0c12a4fed84a5e8b20ad527e07f5ac
I20260929 18:36:18.972657 8911 ts_tablet_manager.cc:1939] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 91f661e2f82c4ae6aeda423adbf82684: Deleting consensus metadata
I20260929 18:36:18.972713 8910 ts_tablet_manager.cc:1918] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:36:18.972787 8910 log.cc:1196] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/2a0c12a4fed84a5e8b20ad527e07f5ac
I20260929 18:36:18.972988 8910 ts_tablet_manager.cc:1939] T 2a0c12a4fed84a5e8b20ad527e07f5ac P 90486608c3ef4ead8332f497801b318d: Deleting consensus metadata
I20260929 18:36:18.973439 8314 catalog_manager.cc:4841] TS 91f661e2f82c4ae6aeda423adbf82684 (127.0.198.2:44965): tablet 2a0c12a4fed84a5e8b20ad527e07f5ac (table test-table [id=d16ae230dcd443158f7172f712238a1d]) successfully deleted
I20260929 18:36:18.973657 8312 catalog_manager.cc:4841] TS 90486608c3ef4ead8332f497801b318d (127.0.198.3:41059): tablet 2a0c12a4fed84a5e8b20ad527e07f5ac (table test-table [id=d16ae230dcd443158f7172f712238a1d]) successfully deleted
W20260929 18:36:18.976487 8315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:39235 (error 108)
I20260929 18:36:18.977284 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 8540
I20260929 18:36:18.977700 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.977261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:39235 (local address 127.0.0.1:33434)
0929 18:36:18.977531 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.977603 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:39235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
W20260929 18:36:18.978643 8315 catalog_manager.cc:4584] TS 360e4436f64d4143b1fb23fe26ce0e8d (127.0.198.1:39235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a0c12a4fed84a5e8b20ad527e07f5ac: Network error: Client connection negotiation failed: client connection to 127.0.198.1:39235: connect: Connection refused (error 111)
I20260929 18:36:18.984179 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 8680
I20260929 18:36:18.989955 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 8296
2026-09-29T18:36:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3510 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:19 dist-test-slave-q5qx krb5kdc[8948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:19 dist-test-slave-q5qx krb5kdc[8948](info): set up 2 sockets
Sep 29 18:36:19 dist-test-slave-q5qx krb5kdc[8948](info): commencing operation
krb5kdc: starting...
W20260929 18:36:19.974363 792 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:20 dist-test-slave-q5qx krb5kdc[8948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:36:20.098012 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:33241
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:33241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:20.207621 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:20.207927 8961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:20.207983 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:20.212211 8961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:20.212291 8961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:20.212316 8961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:20.212337 8961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:20.212355 8961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:20.217891 8961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:33241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:33241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:20.219013 8961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:20.220058 8961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:20.228693 8966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:20.228768 8967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:20.228840 8961 server_base.cc:1034] running on GCE node
W20260929 18:36:20.228704 8969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:20.229580 8961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:20.230835 8961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:20.232043 8961 hybrid_clock.cc:648] HybridClock initialized: now 1790706980232014 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T18:36:20Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1246 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:20 dist-test-slave-q5qx krb5kdc[8974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:20 dist-test-slave-q5qx krb5kdc[8974](info): set up 2 sockets
Sep 29 18:36:20 dist-test-slave-q5qx krb5kdc[8974](info): commencing operation
krb5kdc: starting...
W20260929 18:36:21.194370 792 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:21 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:36:21.317441 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:37695
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:37695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:21.434732 8987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:21.435025 8987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:21.435081 8987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:21.439172 8987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:21.439240 8987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:21.439265 8987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:21.439287 8987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:21.439316 8987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:21.444789 8987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:37695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:37695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:21.445850 8987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:21.446920 8987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:21.455328 8992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:21.455359 8993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:21.455376 8995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:21.455746 8987 server_base.cc:1034] running on GCE node
I20260929 18:36:21.456349 8987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:21.457476 8987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:21.458654 8987 hybrid_clock.cc:648] HybridClock initialized: now 1790706981458640 us; error 27 us; skew 500 ppm
Sep 29 18:36:21 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:21.461520 8987 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:36:21.462479 8987 webserver.cc:460] Webserver started at http://127.0.198.62:42581/ using document root <none> and password file <none>
I20260929 18:36:21.463037 8987 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:21.463105 8987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:21.463296 8987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:21.465363 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "6677d53d62844f17a734f82961c18663"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "926660972930cbdd5b54fd0a62669532"
server_key_iv: "2782d01219301b1b0f952c9ecc951f25"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.465901 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "6677d53d62844f17a734f82961c18663"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "926660972930cbdd5b54fd0a62669532"
server_key_iv: "2782d01219301b1b0f952c9ecc951f25"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.469724 8987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:21.472240 9002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:21.473656 8987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:21.473803 8987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "6677d53d62844f17a734f82961c18663"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "926660972930cbdd5b54fd0a62669532"
server_key_iv: "2782d01219301b1b0f952c9ecc951f25"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.473908 8987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:21.491406 8987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:21.494488 8987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:21.494676 8987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:21.501324 8987 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:37695
I20260929 18:36:21.501358 9054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:37695 every 8 connection(s)
I20260929 18:36:21.502319 8987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:21.504971 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 8987
I20260929 18:36:21.505101 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:21.505292 9055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:21.505401 792 external_mini_cluster.cc:1428] Setting key b84c4abd031ae1f7717ed720484cbf18
I20260929 18:36:21.512743 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Bootstrap starting.
Sep 29 18:36:21 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:21.515431 9055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:21.516299 9055 log.cc:824] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:21.518720 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: No bootstrap required, opened a new log
I20260929 18:36:21.521341 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:21.506810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.0.198.62:37695)
0929 18:36:21.507242 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:21.507263 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:21.507304 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:36:21.508135 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:21.508162 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:21.508616 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:21.509011 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:21.509851 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:21.511396 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:21.512351 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:21.513053 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:21.513219 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:21.515928 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:21.515935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:21.515952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:21.515984 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:21.518056 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:21.518478 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:21.518484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:21.518490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:21.518622 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:21.518947 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:21.518950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:21.518951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:21.519461 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:21.519645 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:21.519931 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:36:21.520286 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
I20260929 18:36:21.523142 9055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:21.523375 9055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:21.523463 9055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6677d53d62844f17a734f82961c18663, State: Initialized, Role: FOLLOWER
I20260929 18:36:21.523936 9055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:21.524066 9055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:21.524125 9055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:21.524216 9055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:21.525111 9055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:21.525434 9055 leader_election.cc:302] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6677d53d62844f17a734f82961c18663; no voters:
I20260929 18:36:21.525744 9055 leader_election.cc:288] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:21.525883 9060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:21.526124 9060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 LEADER]: Becoming Leader. State: Replica: 6677d53d62844f17a734f82961c18663, State: Running, Role: LEADER
I20260929 18:36:21.526448 9060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:21.526961 9055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:21.528107 9061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6677d53d62844f17a734f82961c18663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } } }
I20260929 18:36:21.528108 9062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6677d53d62844f17a734f82961c18663. Latest consensus state: current_term: 1 leader_uuid: "6677d53d62844f17a734f82961c18663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } } }
I20260929 18:36:21.528281 9061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:21.528281 9062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:21.528651 9070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:21.532051 9070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:21.537415 9070 catalog_manager.cc:1347] Generated new cluster ID: c7d8d14de2184c6fb25cf7e3feeb2f83
I20260929 18:36:21.537500 9070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:21.553830 9070 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:21.555212 9070 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:21.564172 9070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Generated new TSK 0
I20260929 18:36:21.565163 9070 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:36:21.597960 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37695
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:21.710379 9081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:21.710734 9081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:21.710803 9081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:21.714900 9081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:21.714962 9081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:21.715044 9081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:21.720387 9081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:21.721489 9081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:21.722570 9081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:21.732421 9081 server_base.cc:1034] running on GCE node
W20260929 18:36:21.732378 9087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:21.732383 9089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:21.732383 9086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:21.733093 9081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:21.733786 9081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:21.735083 9081 hybrid_clock.cc:648] HybridClock initialized: now 1790706981735062 us; error 54 us; skew 500 ppm
Sep 29 18:36:21 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:21.742642 9081 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:36:21.743587 9081 webserver.cc:460] Webserver started at http://127.0.198.1:33039/ using document root <none> and password file <none>
I20260929 18:36:21.744138 9081 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:21.744200 9081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:21.744388 9081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:21.746526 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "3c1b769a8a8549f19e3df33892f3c29a"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "1f4dc7cae555d4fad71cf90d20fb1f3f"
server_key_iv: "ccb8b6241b9da1e7e1eaabb4ad1e2c16"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.747054 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "3c1b769a8a8549f19e3df33892f3c29a"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "1f4dc7cae555d4fad71cf90d20fb1f3f"
server_key_iv: "ccb8b6241b9da1e7e1eaabb4ad1e2c16"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.750682 9081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:21.753232 9096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:21.754611 9081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:21.754719 9081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "3c1b769a8a8549f19e3df33892f3c29a"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "1f4dc7cae555d4fad71cf90d20fb1f3f"
server_key_iv: "ccb8b6241b9da1e7e1eaabb4ad1e2c16"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.754835 9081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:21.765964 9081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:21.769244 9081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:21.769443 9081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:21.770094 9081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:21.770917 9081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:21.770962 9081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:21.771005 9081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:21.771034 9081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:21.780572 9081 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:40645
I20260929 18:36:21.780634 9209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:40645 every 8 connection(s)
I20260929 18:36:21.781503 9081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:21.784709 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 9081
I20260929 18:36:21.784847 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:21.785172 792 external_mini_cluster.cc:1428] Setting key 3567ede0cf7ffed0fd36d3270ad13515
Sep 29 18:36:21 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:36:21.796460 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:21.783411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:53089 (local address 127.0.198.62:37695)
0929 18:36:21.783580 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:21.783585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:21.784528 (+ 943us) server_negotiation.cc:407] Connection header received
0929 18:36:21.786216 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:21.786219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:21.786262 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:21.786346 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:21.787895 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:21.788585 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:21.789737 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:21.789964 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:21.790096 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:21.792795 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:21.792798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:21.792800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:21.792825 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:21.794376 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:21.794923 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:21.794926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:21.794927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:21.795009 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:21.795353 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:21.795356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:21.795357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:21.795555 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:21.795664 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:21.796183 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 18:36:21.796315 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:36:21.797273 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:21.783870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:53089)
0929 18:36:21.784354 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:21.784399 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:21.786013 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:21.786513 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:21.786567 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:21.787041 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:21.787729 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:21.787748 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:21.788715 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:21.788717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:21.789609 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:21.789617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:21.790141 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:21.790144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:21.790318 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:21.790909 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:21.790926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:21.792579 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:21.794499 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:21.794504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:21.794520 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:21.794782 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:21.795101 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:21.795104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:21.795105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:21.795247 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:21.795663 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:21.795669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:21.795936 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 18:36:21.796106 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 18:36:21.796384 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:36:21.798799 9210 heartbeater.cc:347] Connected to a master server at 127.0.198.62:37695
I20260929 18:36:21.799050 9210 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:21.799664 9210 heartbeater.cc:511] Master 127.0.198.62:37695 requested a full tablet report, sending...
I20260929 18:36:21.801322 9019 ts_manager.cc:194] Registered new tserver with Master: 3c1b769a8a8549f19e3df33892f3c29a (127.0.198.1:40645)
I20260929 18:36:21.802587 9019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:53089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:36:21.813717 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37695
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:21.925562 9215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:21.925859 9215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:21.925918 9215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:21.930405 9215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:21.930480 9215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:21.930562 9215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:21.936184 9215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:21.937292 9215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:21.938374 9215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:21.948051 9221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:21.948123 9220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:21.948230 9215 server_base.cc:1034] running on GCE node
W20260929 18:36:21.948123 9223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:21.948837 9215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:21.949486 9215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:21.950667 9215 hybrid_clock.cc:648] HybridClock initialized: now 1790706981950646 us; error 31 us; skew 500 ppm
Sep 29 18:36:21 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:21.953449 9215 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:36:21.954420 9215 webserver.cc:460] Webserver started at http://127.0.198.2:38163/ using document root <none> and password file <none>
I20260929 18:36:21.954959 9215 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:21.955022 9215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:21.955207 9215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:21.957221 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "dfec3991f9a74713a89d40273b077da7"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "21cad5564f7a49de8ca2a2bac24b4519"
server_key_iv: "c8fc1c846f149f91f7ef233722bea078"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.957742 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "dfec3991f9a74713a89d40273b077da7"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "21cad5564f7a49de8ca2a2bac24b4519"
server_key_iv: "c8fc1c846f149f91f7ef233722bea078"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.961428 9215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:21.963999 9230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:21.965236 9215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:21.965335 9215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "dfec3991f9a74713a89d40273b077da7"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "21cad5564f7a49de8ca2a2bac24b4519"
server_key_iv: "c8fc1c846f149f91f7ef233722bea078"
server_key_version: "encryptionkey@0"
I20260929 18:36:21.965451 9215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:21.986397 9215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:21.989624 9215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:21.989800 9215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:21.990478 9215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:21.991325 9215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:21.991374 9215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:21.991412 9215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:21.991428 9215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.000217 9215 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42059
I20260929 18:36:22.000231 9343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42059 every 8 connection(s)
I20260929 18:36:22.001123 9215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:22.010588 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 9215
I20260929 18:36:22.010721 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:22.011010 792 external_mini_cluster.cc:1428] Setting key 0be0ff7c655063f4a6888890e8616f33
I20260929 18:36:22.015594 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.002932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:43563 (local address 127.0.198.62:37695)
0929 18:36:22.003143 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.003147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.004143 (+ 996us) server_negotiation.cc:407] Connection header received
0929 18:36:22.005608 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.005611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.005670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.005742 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.006993 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.007516 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.008669 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.008890 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.008986 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.011568 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:22.011572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:22.011574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:22.011599 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:22.013052 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:22.013735 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:22.013738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:22.013739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:22.013826 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:22.014253 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:22.014256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:22.014258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:22.014607 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:22.014718 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:22.015293 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 18:36:22.015429 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 18:36:22.016418 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.003439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:43563)
0929 18:36:22.003982 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.004025 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.005299 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.005929 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.005937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.006328 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.006838 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.006852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.007634 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.007637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.008549 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.008559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.009041 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.009046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.009210 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.009775 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.009792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.011344 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:22.013181 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:22.013188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:22.013207 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:22.013613 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:22.013921 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:22.013923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:22.013925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:22.014151 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:22.014744 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:22.014750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:22.015014 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 18:36:22.015198 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:36:22.015473 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
I20260929 18:36:22.017845 9344 heartbeater.cc:347] Connected to a master server at 127.0.198.62:37695
I20260929 18:36:22.018170 9344 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:22.018813 9344 heartbeater.cc:511] Master 127.0.198.62:37695 requested a full tablet report, sending...
I20260929 18:36:22.020087 9019 ts_manager.cc:194] Registered new tserver with Master: dfec3991f9a74713a89d40273b077da7 (127.0.198.2:42059)
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:36:22.020777 9019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:43563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:36:22.041227 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37695
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:22.151320 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:22.151585 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:22.151628 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:22.155676 9349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:22.155743 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:22.155803 9349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:22.161161 9349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:37695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:22.162257 9349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:22.163255 9349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:22.173054 9355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:22.173079 9354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:22.173056 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:22.173213 9349 server_base.cc:1034] running on GCE node
I20260929 18:36:22.173810 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:22.174500 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:22.175664 9349 hybrid_clock.cc:648] HybridClock initialized: now 1790706982175635 us; error 42 us; skew 500 ppm
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:22.178589 9349 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:36:22.179517 9349 webserver.cc:460] Webserver started at http://127.0.198.3:43031/ using document root <none> and password file <none>
I20260929 18:36:22.180063 9349 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:22.180123 9349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:22.180318 9349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:22.182415 9349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "1acfe28cba0f4232bbf662700c267faf"
format_stamp: "Formatted at 2026-09-29 18:36:22 on dist-test-slave-q5qx"
server_key: "7b31bb0d8a8d2458ef52c2712c5fb40b"
server_key_iv: "0e1f596daaab84622397443c09cba763"
server_key_version: "encryptionkey@0"
I20260929 18:36:22.182947 9349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "1acfe28cba0f4232bbf662700c267faf"
format_stamp: "Formatted at 2026-09-29 18:36:22 on dist-test-slave-q5qx"
server_key: "7b31bb0d8a8d2458ef52c2712c5fb40b"
server_key_iv: "0e1f596daaab84622397443c09cba763"
server_key_version: "encryptionkey@0"
I20260929 18:36:22.186650 9349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:22.189152 9364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.190578 9349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:22.190690 9349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "1acfe28cba0f4232bbf662700c267faf"
format_stamp: "Formatted at 2026-09-29 18:36:22 on dist-test-slave-q5qx"
server_key: "7b31bb0d8a8d2458ef52c2712c5fb40b"
server_key_iv: "0e1f596daaab84622397443c09cba763"
server_key_version: "encryptionkey@0"
I20260929 18:36:22.190801 9349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:22.219738 9349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:22.223049 9349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:22.223225 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:22.223824 9349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:22.224684 9349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:22.224735 9349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.224790 9349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:22.224820 9349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.233624 9349 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:34279
I20260929 18:36:22.233647 9477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:34279 every 8 connection(s)
I20260929 18:36:22.234573 9349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:22.237739 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 9349
I20260929 18:36:22.237876 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:22.238179 792 external_mini_cluster.cc:1428] Setting key 511b9127a0a70e72c578e85b06759e21
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:22.249213 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.236504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:56993 (local address 127.0.198.62:37695)
0929 18:36:22.236679 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.236683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.237595 (+ 912us) server_negotiation.cc:407] Connection header received
0929 18:36:22.239195 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.239199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.239252 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.239332 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.240858 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.241406 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.242647 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.242864 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.242971 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.245509 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:22.245512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:22.245514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:22.245539 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:22.246932 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:22.247614 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:22.247618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:22.247619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:22.247706 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:22.248045 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:22.248047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:22.248048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:22.248237 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:22.248319 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:22.248958 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 18:36:22.249088 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:36:22.250087 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.236956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:56993)
0929 18:36:22.237412 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.237477 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.238979 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.239597 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.239604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.240124 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.240724 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.240740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.241524 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.241528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.242519 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.242529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.243048 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.243052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.243204 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.243805 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.243826 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.245327 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:22.247089 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:22.247096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:22.247123 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:22.247484 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:22.247802 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:22.247805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:22.247806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:22.247968 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:22.248351 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:22.248357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:22.248692 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 18:36:22.248876 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:36:22.249157 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260929 18:36:22.251413 9478 heartbeater.cc:347] Connected to a master server at 127.0.198.62:37695
I20260929 18:36:22.251766 9478 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:22.252327 9478 heartbeater.cc:511] Master 127.0.198.62:37695 requested a full tablet report, sending...
I20260929 18:36:22.253407 9019 ts_manager.cc:194] Registered new tserver with Master: 1acfe28cba0f4232bbf662700c267faf (127.0.198.3:34279)
I20260929 18:36:22.253947 9019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:56993
I20260929 18:36:22.264074 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab.
Entry for principal oryx/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab.
WARNING: no policy specified for oryx/127.0.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab.
Entry for principal oryx/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab.
WARNING: no policy specified for oryx/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab.
Entry for principal oryx/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab.
I20260929 18:36:22.337199 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 9081
I20260929 18:36:22.342585 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 9215
I20260929 18:36:22.347191 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 9349
I20260929 18:36:22.351671 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 8987
I20260929 18:36:22.356863 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:37695
--webserver_interface=127.0.198.62
--webserver_port=42581
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:37695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:22.467226 9487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:22.467505 9487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:22.467545 9487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:22.471680 9487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:22.471755 9487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:22.471773 9487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:22.471789 9487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:22.471803 9487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:22.477247 9487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:37695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:37695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:22.478381 9487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:22.479393 9487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:22.487990 9487 server_base.cc:1034] running on GCE node
W20260929 18:36:22.487941 9493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:22.487922 9495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:22.487943 9492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:22.488807 9487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:22.490012 9487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:22.491194 9487 hybrid_clock.cc:648] HybridClock initialized: now 1790706982491172 us; error 35 us; skew 500 ppm
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:22.493908 9487 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:36:22.494895 9487 webserver.cc:460] Webserver started at http://127.0.198.62:42581/ using document root <none> and password file <none>
I20260929 18:36:22.495436 9487 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:22.495517 9487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:22.499629 9487 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:22.501921 9502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.503320 9487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:22.503485 9487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "6677d53d62844f17a734f82961c18663"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "926660972930cbdd5b54fd0a62669532"
server_key_iv: "2782d01219301b1b0f952c9ecc951f25"
server_key_version: "encryptionkey@0"
I20260929 18:36:22.504012 9487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:22.514770 9487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:22.517864 9487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:22.518066 9487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:22.524917 9487 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:37695
I20260929 18:36:22.524982 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:37695 every 8 connection(s)
I20260929 18:36:22.525955 9487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:22.530313 9555 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:36:22.531971 9555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Bootstrap starting.
I20260929 18:36:22.532971 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 9487
I20260929 18:36:22.533382 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:40645
--local_ip_for_outbound_sockets=127.0.198.1
--tserver_master_addrs=127.0.198.62:37695
--webserver_port=33039
--webserver_interface=127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:36:22.548185 9555 log.cc:824] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:22.555241 9555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:36:22.555593 9555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Bootstrap complete.
I20260929 18:36:22.559679 9555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:22.560094 9555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6677d53d62844f17a734f82961c18663, State: Initialized, Role: FOLLOWER
I20260929 18:36:22.560583 9555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:22.560704 9555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:22.560771 9555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:22.560863 9555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:36:22.562813 9555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:22.563117 9555 leader_election.cc:302] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6677d53d62844f17a734f82961c18663; no voters:
I20260929 18:36:22.563403 9555 leader_election.cc:288] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:36:22.563557 9559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:36:22.563787 9559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [term 2 LEADER]: Becoming Leader. State: Replica: 6677d53d62844f17a734f82961c18663, State: Running, Role: LEADER
I20260929 18:36:22.564117 9559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } }
I20260929 18:36:22.564543 9555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:22.565129 9559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6677d53d62844f17a734f82961c18663. Latest consensus state: current_term: 2 leader_uuid: "6677d53d62844f17a734f82961c18663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } } }
I20260929 18:36:22.565243 9559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:22.565807 9560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6677d53d62844f17a734f82961c18663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6677d53d62844f17a734f82961c18663" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 37695 } } }
I20260929 18:36:22.565903 9560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:22.566592 9566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:22.570420 9566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:22.571046 9566 catalog_manager.cc:1259] Loaded cluster ID: c7d8d14de2184c6fb25cf7e3feeb2f83
I20260929 18:36:22.571105 9566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:22.573509 9566 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:22.574203 9566 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6677d53d62844f17a734f82961c18663: Loaded TSK: 0
I20260929 18:36:22.575405 9566 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:36:22.649045 9557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:22.649313 9557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:22.649372 9557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:22.653564 9557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:22.653646 9557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:22.653708 9557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:22.659301 9557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.1.keytab
--principal=oryx/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:40645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=33039
--tserver_master_addrs=127.0.198.62:37695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:22.660355 9557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:22.661350 9557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:22.672760 9581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:22.672824 9582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:22.673130 9557 server_base.cc:1034] running on GCE node
W20260929 18:36:22.672761 9584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:22.673642 9557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:22.674316 9557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:22.675487 9557 hybrid_clock.cc:648] HybridClock initialized: now 1790706982675452 us; error 54 us; skew 500 ppm
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:22.678337 9557 init.cc:377] Logged in from keytab as oryx/127.0.198.1@KRBTEST.COM (short username oryx)
I20260929 18:36:22.679360 9557 webserver.cc:460] Webserver started at http://127.0.198.1:33039/ using document root <none> and password file <none>
I20260929 18:36:22.679909 9557 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:22.679997 9557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:22.684127 9557 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:22.686285 9591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.687768 9557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:22.687897 9557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "3c1b769a8a8549f19e3df33892f3c29a"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "1f4dc7cae555d4fad71cf90d20fb1f3f"
server_key_iv: "ccb8b6241b9da1e7e1eaabb4ad1e2c16"
server_key_version: "encryptionkey@0"
I20260929 18:36:22.688566 9557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:22.719928 9557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:22.723254 9557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:22.723434 9557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:22.724074 9557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:22.724937 9557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:22.724987 9557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.725028 9557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:22.725056 9557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.733400 9557 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:40645
I20260929 18:36:22.733475 9704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:40645 every 8 connection(s)
I20260929 18:36:22.734442 9557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:22.740074 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 9557
I20260929 18:36:22.740635 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:42059
--local_ip_for_outbound_sockets=127.0.198.2
--tserver_master_addrs=127.0.198.62:37695
--webserver_port=38163
--webserver_interface=127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:22.749411 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.736596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:46931)
0929 18:36:22.737326 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.737377 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.738669 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.739870 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.739877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.740289 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.740862 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.740879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.742662 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.742666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.743786 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.743798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.744708 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.744713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.744900 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.745590 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.745608 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.748297 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":134,"threads_started":1}
W20260929 18:36:22.749400 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:22.736552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:46931 (local address 127.0.198.62:37695)
0929 18:36:22.737030 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.737050 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.737519 (+ 469us) server_negotiation.cc:407] Connection header received
0929 18:36:22.739095 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.739108 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.739444 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.739753 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.741083 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.742469 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.743945 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.744498 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.744683 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.748647 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:46931: BlockingRecv error: recv got EOF from 127.0.198.1:46931 (error 108)
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":101,"threads_started":1}
W20260929 18:36:22.749993 9705 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:37695 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.198.62:37695: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:22.756376 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.750484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:54789)
0929 18:36:22.750727 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.750739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.750822 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.751124 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.751127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.751270 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.751394 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.751400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.752488 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.752491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.753068 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.753077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.753626 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.753629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.753705 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.754245 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.754257 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.756258 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260929 18:36:22.756686 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:22.750555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:54789 (local address 127.0.198.62:37695)
0929 18:36:22.750656 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.750660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.750814 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:36:22.750904 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.750907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.750968 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.751059 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.751509 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.752352 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.753173 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.753517 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.753609 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.756442 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:54789: BlockingRecv error: recv got EOF from 127.0.198.1:54789 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:22.762354 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.756936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:58329)
0929 18:36:22.757097 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.757109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.757190 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.757443 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.757446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.757575 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.757648 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.757652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.758699 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.758702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.759236 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.759246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.759787 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.759790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.759853 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.760325 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.760336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.762256 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 18:36:22.762513 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:22.756962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:58329 (local address 127.0.198.62:37695)
0929 18:36:22.757054 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.757058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.757169 (+ 111us) server_negotiation.cc:407] Connection header received
0929 18:36:22.757267 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.757270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.757320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.757397 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.757742 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.758575 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.759337 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.759669 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.759768 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.762408 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:58329: BlockingRecv error: recv got EOF from 127.0.198.1:58329 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260929 18:36:22.762723 9705 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:36:22.856968 9709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:22.857239 9709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:22.857286 9709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:22.861639 9709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:22.861734 9709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:22.861799 9709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:22.867475 9709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.2.keytab
--principal=oryx/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:42059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=38163
--tserver_master_addrs=127.0.198.62:37695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:22.868672 9709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:22.869671 9709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:22.879406 9714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:22.879427 9717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:22.879465 9709 server_base.cc:1034] running on GCE node
W20260929 18:36:22.879555 9715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:22.880113 9709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:22.880764 9709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:22.881933 9709 hybrid_clock.cc:648] HybridClock initialized: now 1790706982881911 us; error 34 us; skew 500 ppm
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:22.884759 9709 init.cc:377] Logged in from keytab as oryx/127.0.198.2@KRBTEST.COM (short username oryx)
I20260929 18:36:22.885696 9709 webserver.cc:460] Webserver started at http://127.0.198.2:38163/ using document root <none> and password file <none>
I20260929 18:36:22.886335 9709 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:22.886410 9709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:22.890530 9709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:22.892743 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.894181 9709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:22.894301 9709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "dfec3991f9a74713a89d40273b077da7"
format_stamp: "Formatted at 2026-09-29 18:36:21 on dist-test-slave-q5qx"
server_key: "21cad5564f7a49de8ca2a2bac24b4519"
server_key_iv: "c8fc1c846f149f91f7ef233722bea078"
server_key_version: "encryptionkey@0"
I20260929 18:36:22.894843 9709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:22.926630 9709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:22.929773 9709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:22.929991 9709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:22.930625 9709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:22.931452 9709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:22.931505 9709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.931545 9709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:22.931573 9709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:22.940368 9709 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42059
I20260929 18:36:22.940394 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42059 every 8 connection(s)
I20260929 18:36:22.941378 9709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:22.948657 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 9709
I20260929 18:36:22.949095 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:34279
--local_ip_for_outbound_sockets=127.0.198.3
--tserver_master_addrs=127.0.198.62:37695
--webserver_port=43031
--webserver_interface=127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:36:22.953366 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:22.943275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:55987 (local address 127.0.198.62:37695)
0929 18:36:22.943528 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.943533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.944361 (+ 828us) server_negotiation.cc:407] Connection header received
0929 18:36:22.945730 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.945733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.945785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.945865 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.947180 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.947750 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.949045 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.949284 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.949384 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.953240 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:55987: BlockingRecv error: recv got EOF from 127.0.198.2:55987 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
I20260929 18:36:22.954093 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.943737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:55987)
0929 18:36:22.944206 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.944248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.945498 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.946102 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.946109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.946496 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.947015 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.947030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.947881 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.947885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.948912 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.948924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.949434 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.949438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.949624 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.950239 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.950262 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.953102 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
W20260929 18:36:22.954605 9838 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:37695 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.198.62:37695: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:22.959972 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.955077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:33565)
0929 18:36:22.955190 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.955205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.955294 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.955587 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.955590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.955730 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.955824 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.955829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.956547 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.956549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.957103 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.957112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.957547 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.957550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.957641 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.958121 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.958132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.959872 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 18:36:22.960145 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:22.955165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:33565 (local address 127.0.198.62:37695)
0929 18:36:22.955274 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.955278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.955291 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:22.955389 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.955392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.955448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.955511 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.955931 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.956445 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.957212 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.957450 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.957534 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.960030 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:33565: BlockingRecv error: recv got EOF from 127.0.198.2:33565 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:22 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:22.966071 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:22.960731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:48599)
0929 18:36:22.960834 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:22.960849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:22.960927 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:22.961146 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:22.961148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:22.961271 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:22.961345 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.961352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.962411 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.962413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.962995 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:22.963004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.963528 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.963530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:22.963587 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.964040 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:22.964058 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:22.965916 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:36:22.966208 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:22.960828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:48599 (local address 127.0.198.62:37695)
0929 18:36:22.960959 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:22.960963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:22.960977 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:22.961015 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:22.961018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:22.961059 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:22.961134 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:22.961434 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.962267 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.963092 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:22.963427 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:22.963518 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:22.966089 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:48599: BlockingRecv error: recv got EOF from 127.0.198.2:48599 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 18:36:22.966472 9838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:36:23.062083 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:23.062383 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:23.062446 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:23.067250 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:23.067324 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:23.067406 9841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:23.073390 9841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/krb5kdc/oryx_127.0.198.3.keytab
--principal=oryx/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:34279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=43031
--tserver_master_addrs=127.0.198.62:37695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:23.074616 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:23.075977 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:23.086225 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:23.086227 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:23.086365 9841 server_base.cc:1034] running on GCE node
W20260929 18:36:23.086558 9849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:23.087007 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:23.087647 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:23.088806 9841 hybrid_clock.cc:648] HybridClock initialized: now 1790706983088797 us; error 46 us; skew 500 ppm
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:23.091702 9841 init.cc:377] Logged in from keytab as oryx/127.0.198.3@KRBTEST.COM (short username oryx)
I20260929 18:36:23.092679 9841 webserver.cc:460] Webserver started at http://127.0.198.3:43031/ using document root <none> and password file <none>
I20260929 18:36:23.093209 9841 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:23.093277 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:23.097386 9841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:23.099620 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:23.100857 9841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:23.100970 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "1acfe28cba0f4232bbf662700c267faf"
format_stamp: "Formatted at 2026-09-29 18:36:22 on dist-test-slave-q5qx"
server_key: "7b31bb0d8a8d2458ef52c2712c5fb40b"
server_key_iv: "0e1f596daaab84622397443c09cba763"
server_key_version: "encryptionkey@0"
I20260929 18:36:23.101570 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:23.123198 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:23.126327 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:23.126518 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:23.127090 9841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:23.127892 9841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:23.127944 9841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:23.127995 9841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:23.128021 9841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:23.136471 9841 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:34279
I20260929 18:36:23.136515 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:34279 every 8 connection(s)
I20260929 18:36:23.137557 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:23.146517 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 9841
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:36:23.149428 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:23.139414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:52679 (local address 127.0.198.62:37695)
0929 18:36:23.139560 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.139565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.140433 (+ 868us) server_negotiation.cc:407] Connection header received
0929 18:36:23.141821 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.141825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.141877 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.141957 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.143202 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.143740 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.144961 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.145194 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.145278 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.149328 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:52679: BlockingRecv error: recv got EOF from 127.0.198.3:52679 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:36:23.150152 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.139809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:52679)
0929 18:36:23.140278 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:23.140320 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:23.141568 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:23.142170 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:23.142177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:23.142556 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:23.143043 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.143057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.143908 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.143911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.144827 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.144844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.145348 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.145351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.145520 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.146137 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:23.146160 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:23.149234 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
W20260929 18:36:23.150635 9970 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:37695 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.198.62:37695: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:23.156342 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.151052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:52919)
0929 18:36:23.151152 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:23.151165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:23.151231 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:23.151469 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:23.151472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:23.151634 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:23.151732 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.151737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.152658 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.152661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.153198 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.153206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.153607 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.153610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.153697 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.154125 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:23.154136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:23.156214 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:36:23.156458 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:23.151114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:52919 (local address 127.0.198.62:37695)
0929 18:36:23.151213 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.151216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.151227 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:36:23.151315 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.151318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.151361 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.151423 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.151833 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.152567 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.153300 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.153519 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.153599 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.156352 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:52919: BlockingRecv error: recv got EOF from 127.0.198.3:52919 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:36:23.156715 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.147571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.0.198.62:37695)
0929 18:36:23.147787 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.147791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.147803 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:23.147840 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.147843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.147886 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.147994 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.148796 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.149550 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.150533 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.150840 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.150929 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.152538 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:23.152543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:23.152558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:23.152584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:23.154262 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:23.154750 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:23.154757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:23.154764 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:23.154934 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:23.155227 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:23.155229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:23.155230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:23.155881 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:23.156064 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:23.156354 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:36:23.156593 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:23.162109 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.157134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:45081)
0929 18:36:23.157355 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:23.157366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:23.157430 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:23.157643 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:23.157645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:23.157758 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:23.157825 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.157829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.158890 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.158892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.159558 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.159567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.159975 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.159978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.160061 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.160456 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:23.160469 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:23.161961 (+ 1492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160}
W20260929 18:36:23.162206 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:23.157231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:45081 (local address 127.0.198.62:37695)
0929 18:36:23.157363 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.157368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.157402 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:36:23.157507 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.157510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.157548 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.157622 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.157912 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.158600 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.159656 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.159885 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.159964 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.162113 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:45081: BlockingRecv error: recv got EOF from 127.0.198.3:45081 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 18:36:23.162328 9970 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:23.769047 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.763438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:60253)
0929 18:36:23.763690 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:23.763704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:23.763803 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:23.764116 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:23.764119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:23.764326 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:23.764470 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.764478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.765437 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.765441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.766051 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.766063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.766623 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.766627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.766710 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.767215 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:23.767227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:23.768919 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
W20260929 18:36:23.769189 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:23.763575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:60253 (local address 127.0.198.62:37695)
0929 18:36:23.763849 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.763853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.763865 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:23.763903 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.763905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.763945 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.764050 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.764611 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.765325 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.766175 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.766509 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.766599 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.769069 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:60253: BlockingRecv error: recv got EOF from 127.0.198.1:60253 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:23 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:23.972285 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.967116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:47833)
0929 18:36:23.967373 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:23.967386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:23.967495 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:23.967727 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:23.967730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:23.967927 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:23.968017 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.968023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.968797 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.968800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.969372 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.969380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.969791 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.969795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:23.969884 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.970397 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:23.970411 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:23.972123 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
W20260929 18:36:23.972368 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:23.967184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:47833 (local address 127.0.198.62:37695)
0929 18:36:23.967323 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.967327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.967464 (+ 137us) server_negotiation.cc:407] Connection header received
0929 18:36:23.967580 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.967583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.967629 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.967688 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.968128 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.968686 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.969482 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.969693 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.969776 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.972261 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:47833: BlockingRecv error: recv got EOF from 127.0.198.2:47833 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 18:36:24 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:24 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:24.169263 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.163041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:43595)
0929 18:36:24.163311 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.163332 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.163440 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.163808 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.163810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.163980 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:24.164125 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.164132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.165032 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.165036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.165764 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.165774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.166446 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.166449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.166522 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.167017 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:24.167031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:24.169131 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
W20260929 18:36:24.169457 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:24.163204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:43595 (local address 127.0.198.62:37695)
0929 18:36:24.163423 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.163427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.163444 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:36:24.163587 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.163589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.163652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.163734 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:24.164263 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.164890 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.165958 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.166239 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.166454 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.169279 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:43595: BlockingRecv error: recv got EOF from 127.0.198.3:43595 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 18:36:24 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:24 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:24.775440 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.769917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:34093)
0929 18:36:24.770230 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.770243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.770359 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.770605 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.770607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.770775 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:24.770893 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.770898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.771637 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.771639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.772204 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.772211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.772684 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.772687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.772755 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.773273 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:24.773287 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:24.775300 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
W20260929 18:36:24.775575 9980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:24.770028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:34093 (local address 127.0.198.62:37695)
0929 18:36:24.770263 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.770267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.770348 (+ 81us) server_negotiation.cc:407] Connection header received
0929 18:36:24.770457 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.770460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.770502 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.770579 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:24.770988 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.771549 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.772352 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.772576 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.772657 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.775444 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:34093: BlockingRecv error: recv got EOF from 127.0.198.1:34093 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
Sep 29 18:36:24 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:24 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:24.978488 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.973168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:44089)
0929 18:36:24.973454 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.973467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.973573 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.973886 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.973888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.974090 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:24.974188 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.974199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.974983 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.974992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.975560 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.975574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.975976 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.975979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.976048 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.976528 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:24.976540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:24.978355 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
W20260929 18:36:24.978626 9980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:24.973269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:44089 (local address 127.0.198.62:37695)
0929 18:36:24.973422 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.973427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.973533 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:36:24.973686 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.973691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.973741 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.973819 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:24.974309 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.974867 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.975656 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.975863 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.975934 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.978505 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:44089: BlockingRecv error: recv got EOF from 127.0.198.2:44089 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 18:36:25 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:25 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:25.175216 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.170148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:54249)
0929 18:36:25.170423 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.170438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.170545 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.170792 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.170795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.170981 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:25.171069 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.171075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.171797 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.171800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.172342 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.172350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.172769 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.172772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.172849 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.173316 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:25.173327 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:25.175039 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260929 18:36:25.175343 9980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:25.170312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:54249 (local address 127.0.198.62:37695)
0929 18:36:25.170430 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.170435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.170501 (+ 66us) server_negotiation.cc:407] Connection header received
0929 18:36:25.170634 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.170637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.170685 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.170754 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:25.171177 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.171697 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.172448 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.172670 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.172753 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.175185 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:54249: BlockingRecv error: recv got EOF from 127.0.198.3:54249 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 18:36:25 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:25 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:25.782459 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.776284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:57089)
0929 18:36:25.776542 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.776556 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.776647 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.776896 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.776898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.777075 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:25.777166 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.777171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.777997 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.778000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.778560 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.778567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.779082 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.779087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.779176 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.779908 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:25.779930 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:25.782323 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
W20260929 18:36:25.782560 9984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:25.776394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:57089 (local address 127.0.198.62:37695)
0929 18:36:25.776644 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.776647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.776663 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:36:25.776739 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.776742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.776783 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.776854 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:25.777318 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.777872 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.778689 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.778944 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.779042 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.782450 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:57089: BlockingRecv error: recv got EOF from 127.0.198.1:57089 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
Sep 29 18:36:25 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:25 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:25.985157 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.979435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:37245)
0929 18:36:25.979699 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.979712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.979810 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.980159 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.980161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.980369 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:25.980468 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.980474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.981223 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.981225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.981835 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.981844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.982444 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.982448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.982537 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.983076 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:25.983087 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:25.984981 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
W20260929 18:36:25.985229 9984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:25.979558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:37245 (local address 127.0.198.62:37695)
0929 18:36:25.979718 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.979721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.979778 (+ 57us) server_negotiation.cc:407] Connection header received
0929 18:36:25.980000 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.980003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.980054 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.980134 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:25.980591 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.981116 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.981964 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.982293 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.982385 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.985131 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:37245: BlockingRecv error: recv got EOF from 127.0.198.2:37245 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 18:36:26 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:26 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:26.180966 9986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:26.175965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:35483)
0929 18:36:26.176215 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:26.176229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:26.176323 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:26.176541 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:26.176543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:26.176710 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:26.176812 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:26.176818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.177540 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.177542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:26.178137 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:26.178145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.178550 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.178552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:26.178613 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:26.179099 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:26.179111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:26.180830 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
W20260929 18:36:26.181088 9984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:26.176062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:35483 (local address 127.0.198.62:37695)
0929 18:36:26.176232 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:26.176235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:26.176278 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:36:26.176399 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:26.176401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:26.176454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:26.176512 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:26.176937 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.177460 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.178231 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.178457 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.178538 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:26.180971 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:35483: BlockingRecv error: recv got EOF from 127.0.198.3:35483 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 18:36:26 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:26 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:26.793816 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:26.783168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:33901)
0929 18:36:26.783425 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:26.783438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:26.783527 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:26.783764 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:26.783767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:26.783937 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:26.784028 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:26.784033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.784780 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.784783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:26.785333 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:26.785340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.785790 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.785793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:26.785851 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:26.791578 (+ 5727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:26.791591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:26.793631 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
W20260929 18:36:26.793943 9988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:26.783244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:33901 (local address 127.0.198.62:37695)
0929 18:36:26.783488 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:26.783492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:26.783504 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:26.783610 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:26.783612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:26.783655 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:26.783728 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:26.784129 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.784684 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.785462 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.785695 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.785782 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:26.793809 (+ 8027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:33901: BlockingRecv error: recv got EOF from 127.0.198.1:33901 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
Sep 29 18:36:26 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:26 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:26.991182 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:26.986032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:49495)
0929 18:36:26.986314 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:26.986327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:26.986436 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:26.986679 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:26.986681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:26.986876 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:26.986967 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:26.986973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.987720 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.987723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:26.988273 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:26.988281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.988693 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.988696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:26.988774 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:26.989258 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:26.989270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:26.991053 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
W20260929 18:36:26.991324 9988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:26.986100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:49495 (local address 127.0.198.62:37695)
0929 18:36:26.986268 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:26.986272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:26.986396 (+ 124us) server_negotiation.cc:407] Connection header received
0929 18:36:26.986518 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:26.986522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:26.986569 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:26.986642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:26.987088 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.987619 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.988383 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:26.988597 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:26.988682 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:26.991218 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:49495: BlockingRecv error: recv got EOF from 127.0.198.2:49495 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 18:36:27 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:27 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:27.186976 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.181769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:43027)
0929 18:36:27.182059 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:27.182074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:27.182171 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:27.182399 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:27.182401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:27.182569 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:27.182662 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.182667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.183401 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.183403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.183964 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.183972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.184394 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.184396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.184457 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.184915 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:27.184926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:27.186853 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
W20260929 18:36:27.187122 9988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:27.181855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:43027 (local address 127.0.198.62:37695)
0929 18:36:27.181995 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:27.181999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:27.182135 (+ 136us) server_negotiation.cc:407] Connection header received
0929 18:36:27.182253 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:27.182255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:27.182299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:27.182367 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:27.182758 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.183309 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.184059 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.184302 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.184376 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.187008 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:43027: BlockingRecv error: recv got EOF from 127.0.198.3:43027 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:36:27 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:27 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:27.800383 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.794654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:53815)
0929 18:36:27.794951 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:27.794967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:27.795074 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:27.795351 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:27.795354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:27.795578 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:27.795696 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.795703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.796553 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.796556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.797217 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.797228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.797775 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.797779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.797873 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.798407 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:27.798419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:27.800244 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
W20260929 18:36:27.800518 9992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:27.794770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:53815 (local address 127.0.198.62:37695)
0929 18:36:27.795008 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:27.795012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:27.795025 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:27.795168 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:27.795171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:27.795224 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:27.795317 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:27.795818 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.796426 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.797353 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.797627 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.797741 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.800387 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:53815: BlockingRecv error: recv got EOF from 127.0.198.1:53815 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
Sep 29 18:36:27 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:27 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:27.997395 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.992036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:55445)
0929 18:36:27.992331 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:27.992346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:27.992444 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:27.992681 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:27.992683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:27.992923 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:27.993022 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.993029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.993860 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.993862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.994486 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.994495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.994871 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.994873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.994930 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.995401 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:27.995412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:27.997187 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
W20260929 18:36:27.997435 9992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:27.992213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:55445 (local address 127.0.198.62:37695)
0929 18:36:27.992343 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:27.992346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:27.992407 (+ 61us) server_negotiation.cc:407] Connection header received
0929 18:36:27.992514 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:27.992516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:27.992556 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:27.992623 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:27.993182 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.993728 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.994575 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.994788 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.994863 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.997340 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:55445: BlockingRecv error: recv got EOF from 127.0.198.2:55445 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:36:28 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:28 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:28.193048 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.187797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:51397)
0929 18:36:28.188072 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.188084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.188182 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.188454 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.188456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.188648 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:28.188735 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.188741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.189461 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.189465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.190062 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.190070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.190476 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.190479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.190557 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.191020 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:28.191031 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:28.192935 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
W20260929 18:36:28.193277 9992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:28.187874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:51397 (local address 127.0.198.62:37695)
0929 18:36:28.188006 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.188010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.188146 (+ 136us) server_negotiation.cc:407] Connection header received
0929 18:36:28.188306 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.188309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.188358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.188419 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:28.188844 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.189364 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.190167 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.190382 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.190464 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.193073 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:51397: BlockingRecv error: recv got EOF from 127.0.198.3:51397 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 18:36:28 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:28 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:28.806751 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.801238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:54821)
0929 18:36:28.801513 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.801529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.801664 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.801953 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.801956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.802180 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:28.802273 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.802278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.803022 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.803025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.803581 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.803588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.804021 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.804025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.804104 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.804649 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:28.804665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:28.806590 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
W20260929 18:36:28.806849 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:28.801309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:54821 (local address 127.0.198.62:37695)
0929 18:36:28.801514 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.801519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.801611 (+ 92us) server_negotiation.cc:407] Connection header received
0929 18:36:28.801782 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.801785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.801830 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.801912 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:28.802389 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.802919 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.803696 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.803919 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.804005 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.806738 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:54821: BlockingRecv error: recv got EOF from 127.0.198.1:54821 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":55,"threads_started":1}
Sep 29 18:36:28 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:28 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:29.003670 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.998285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:59605)
0929 18:36:28.998558 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.998573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.998692 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.998951 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.998958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.999152 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:28.999249 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.999255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.000081 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.000083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.000689 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.000698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.001146 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.001149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.001220 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.001739 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:29.001751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:29.003536 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
W20260929 18:36:29.003823 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:28.998360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:59605 (local address 127.0.198.62:37695)
0929 18:36:28.998537 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.998542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.998634 (+ 92us) server_negotiation.cc:407] Connection header received
0929 18:36:28.998762 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.998764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.998807 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.998877 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:28.999353 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.999956 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.000801 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.001021 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.001103 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.003693 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:59605: BlockingRecv error: recv got EOF from 127.0.198.2:59605 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 18:36:29 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:29 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:29.199132 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.193762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:48949)
0929 18:36:29.194051 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.194109 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.194226 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.194481 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.194484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.194681 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:29.194772 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.194779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.195506 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.195509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.196081 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.196090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.196503 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.196507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.196585 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.197089 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:29.197101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:29.198974 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
W20260929 18:36:29.199242 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:29.193865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:48949 (local address 127.0.198.62:37695)
0929 18:36:29.194042 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.194047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.194188 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:36:29.194333 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.194336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.194381 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.194444 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:29.194890 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.195404 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.196192 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.196405 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.196491 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.199144 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:48949: BlockingRecv error: recv got EOF from 127.0.198.3:48949 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:36:29 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:29 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:29.813625 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.807679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:48129)
0929 18:36:29.808027 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.808044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.808189 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.808436 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.808438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.808643 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:29.808736 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.808742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.809627 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.809631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.810419 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.810432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.810985 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.810989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.811079 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.811721 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:29.811733 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:29.813500 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":87,"threads_started":1}
W20260929 18:36:29.813828 10000 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:29.807679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:48129 (local address 127.0.198.62:37695)
0929 18:36:29.808012 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.808016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.808201 (+ 185us) server_negotiation.cc:407] Connection header received
0929 18:36:29.808270 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.808272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.808321 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.808408 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:29.808868 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.809490 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.810581 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.810841 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.810954 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.813689 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:48129: BlockingRecv error: recv got EOF from 127.0.198.1:48129 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
Sep 29 18:36:30 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:30 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:30.010243 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:30.004379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:60175)
0929 18:36:30.004713 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:30.004726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:30.004829 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:30.005177 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:30.005179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:30.005362 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:30.005455 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:30.005461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.006399 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.006402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:30.006995 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:30.007005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.007476 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.007480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:30.007548 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:30.008079 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:30.008092 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:30.010097 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":88,"threads_started":1}
W20260929 18:36:30.010422 10000 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:30.004576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:60175 (local address 127.0.198.62:37695)
0929 18:36:30.004717 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:30.004721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:30.004822 (+ 101us) server_negotiation.cc:407] Connection header received
0929 18:36:30.005013 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:30.005017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:30.005075 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:30.005140 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:30.005607 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.006259 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.007130 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.007347 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.007423 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:30.010296 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:60175: BlockingRecv error: recv got EOF from 127.0.198.2:60175 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:36:30 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:30 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:30.206666 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:30.200046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:35335)
0929 18:36:30.200347 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:30.200360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:30.200473 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:30.200753 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:30.200755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:30.201011 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:30.201113 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:30.201120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.202270 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.202274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:30.202856 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:30.202865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.203471 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.203475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:30.203560 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:30.204324 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:30.204340 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:30.206518 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
W20260929 18:36:30.206840 10000 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:30.200130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:35335 (local address 127.0.198.62:37695)
0929 18:36:30.200276 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:30.200281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:30.200435 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:36:30.200558 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:30.200561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:30.200617 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:30.200720 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:30.201224 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.202112 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.202967 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.203323 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.203436 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:30.206709 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:35335: BlockingRecv error: recv got EOF from 127.0.198.3:35335 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 18:36:30 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:30 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:30.819716 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:30.814338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:40373)
0929 18:36:30.814582 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:30.814595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:30.814679 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:30.814908 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:30.814910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:30.815090 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:30.815186 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:30.815192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.816039 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.816042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:30.816608 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:30.816616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.817094 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.817096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:30.817156 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:30.817628 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:30.817639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:30.819597 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
W20260929 18:36:30.819882 10004 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:30.814488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:40373 (local address 127.0.198.62:37695)
0929 18:36:30.814707 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:30.814711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:30.814723 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:36:30.814755 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:30.814757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:30.814796 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:30.814876 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:30.815328 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.815939 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.816740 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:30.816989 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:30.817076 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:30.819737 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:40373: BlockingRecv error: recv got EOF from 127.0.198.1:40373 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
Sep 29 18:36:31 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:31 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:31.017359 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:31.011151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:46767)
0929 18:36:31.011423 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:31.011435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:31.011552 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:31.011822 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:31.011825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:31.012017 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:31.012116 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.012121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.013102 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.013105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.014095 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.014107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.014791 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.014796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.014894 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.015477 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:31.015489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:31.017240 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
W20260929 18:36:31.017513 10004 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:31.011276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:46767 (local address 127.0.198.62:37695)
0929 18:36:31.011411 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:31.011415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:31.011538 (+ 123us) server_negotiation.cc:407] Connection header received
0929 18:36:31.011625 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:31.011627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:31.011684 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:31.011766 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:31.012224 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.012929 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.014252 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.014629 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.014744 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.017389 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:46767: BlockingRecv error: recv got EOF from 127.0.198.2:46767 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 18:36:31 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:31 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:31.212574 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:31.207333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:41223)
0929 18:36:31.207587 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:31.207599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:31.207696 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:31.207915 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:31.207917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:31.208085 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:31.208175 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.208180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.208975 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.208978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.209529 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.209536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.210091 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.210095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.210163 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.210618 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:31.210630 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:31.212455 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
W20260929 18:36:31.212745 10004 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:31.207453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:41223 (local address 127.0.198.62:37695)
0929 18:36:31.207584 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:31.207588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:31.207659 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:36:31.207765 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:31.207767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:31.207809 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:31.207873 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:31.208273 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.208802 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.209637 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.209849 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.209923 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.212627 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:41223: BlockingRecv error: recv got EOF from 127.0.198.3:41223 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 18:36:31 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:31 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:31.830863 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:31.820740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:56047)
0929 18:36:31.821045 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:31.821059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:31.821152 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:31.821383 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:31.821385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:31.821560 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:31.821654 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.821659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.822430 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.822433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.823057 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.823066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.823484 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.823487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.823554 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.828776 (+ 5222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:31.828794 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:31.830734 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
W20260929 18:36:31.831027 10008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:31.820792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:56047 (local address 127.0.198.62:37695)
0929 18:36:31.821106 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:31.821109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:31.821123 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:31.821238 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:31.821240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:31.821282 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:31.821351 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:31.821741 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.822342 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.823158 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.823380 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.823459 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.830910 (+ 7451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:56047: BlockingRecv error: recv got EOF from 127.0.198.1:56047 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
Sep 29 18:36:32 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:32 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:32.023664 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.018183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:37281)
0929 18:36:32.018461 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.018474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.018568 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.018884 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.018887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.019121 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:32.019231 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.019237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.020010 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.020014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.020598 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.020607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.021097 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.021100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.021169 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.021738 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:32.021750 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:32.023547 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
W20260929 18:36:32.023849 10008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:32.018295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:37281 (local address 127.0.198.62:37695)
0929 18:36:32.018422 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.018425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.018526 (+ 101us) server_negotiation.cc:407] Connection header received
0929 18:36:32.018648 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.018650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.018691 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.018750 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:32.019333 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.019858 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.020732 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.020977 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.021072 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.023720 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:37281: BlockingRecv error: recv got EOF from 127.0.198.2:37281 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 18:36:32 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:32 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:32.218695 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.213398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:59847)
0929 18:36:32.213674 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.213689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.213788 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.214050 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.214053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.214264 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:32.214381 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.214389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.215195 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.215198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.215760 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.215768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.216188 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.216190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.216253 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.216725 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:32.216736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:32.218581 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
W20260929 18:36:32.218817 10008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:32.213458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:59847 (local address 127.0.198.62:37695)
0929 18:36:32.213588 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.213593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.213762 (+ 169us) server_negotiation.cc:407] Connection header received
0929 18:36:32.213881 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.213883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.213932 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.214028 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:32.214515 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.215090 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.215874 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.216089 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.216165 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.218713 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:59847: BlockingRecv error: recv got EOF from 127.0.198.3:59847 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 18:36:32 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:32 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:32.837339 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.831798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:41195)
0929 18:36:32.832064 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.832081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.832184 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.832545 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.832548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.832731 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:32.832833 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.832841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.833654 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.833657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.834292 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.834311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.834791 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.834794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.834866 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.835391 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:32.835403 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:32.837217 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
W20260929 18:36:32.837558 10012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:32.831985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:41195 (local address 127.0.198.62:37695)
0929 18:36:32.832290 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.832294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.832307 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:32.832343 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.832345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.832387 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.832474 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:32.832964 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.833541 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.834426 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.834665 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.834753 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.837403 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:41195: BlockingRecv error: recv got EOF from 127.0.198.1:41195 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":105,"threads_started":1}
Sep 29 18:36:33 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:33 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:33.030033 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.024511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:46323)
0929 18:36:33.024862 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.024874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.024988 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.025227 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.025230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.025422 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:33.025514 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.025520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.026338 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.026341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.026920 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.026928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.027339 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.027342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.027422 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.027957 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:33.027968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:33.029815 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":108,"threads_started":1}
W20260929 18:36:33.030088 10012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:33.024655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:46323 (local address 127.0.198.62:37695)
0929 18:36:33.024801 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.024805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.024954 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:36:33.025077 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.025080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.025126 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.025194 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:33.025627 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.026225 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.027024 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.027239 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.027326 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.029989 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:46323: BlockingRecv error: recv got EOF from 127.0.198.2:46323 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 18:36:33 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:33 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:33.225051 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.219588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:41671)
0929 18:36:33.220019 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.220032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.220121 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.220390 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.220393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.220625 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:33.220734 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.220742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.221467 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.221471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.222082 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.222093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.222553 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.222555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.222618 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.223133 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:33.223143 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:33.224917 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
W20260929 18:36:33.225147 10012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:33.219727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:41671 (local address 127.0.198.62:37695)
0929 18:36:33.219907 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.219910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.220103 (+ 193us) server_negotiation.cc:407] Connection header received
0929 18:36:33.220192 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.220195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.220242 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.220351 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:33.220855 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.221370 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.222197 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.222451 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.222533 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.225055 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:41671: BlockingRecv error: recv got EOF from 127.0.198.3:41671 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 18:36:33 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:33 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:33.843940 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.838204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:54759)
0929 18:36:33.838461 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.838475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.838566 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.838797 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.838799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.838977 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:33.839074 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.839079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.839831 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.839835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.840588 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.840601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.841153 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.841156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.841223 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.841703 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:33.841714 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:33.843711 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
W20260929 18:36:33.843999 10016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:33.838331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:54759 (local address 127.0.198.62:37695)
0929 18:36:33.838584 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.838587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.838602 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:36:33.838644 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.838646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.838686 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.838775 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:33.839169 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.839696 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.840726 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.841037 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.841129 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.843880 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:54759: BlockingRecv error: recv got EOF from 127.0.198.1:54759 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
Sep 29 18:36:34 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:34 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:34.036408 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:34.030835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:42753)
0929 18:36:34.031091 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:34.031103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:34.031202 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:34.031535 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:34.031538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:34.031720 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:34.031820 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:34.031827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.032643 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.032646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:34.033212 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:34.033221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.033702 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.033706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:34.033791 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:34.034331 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:34.034344 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:34.036253 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
W20260929 18:36:34.036527 10016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:34.030882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:42753 (local address 127.0.198.62:37695)
0929 18:36:34.031020 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:34.031024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:34.031170 (+ 146us) server_negotiation.cc:407] Connection header received
0929 18:36:34.031318 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:34.031320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:34.031367 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:34.031453 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:34.031962 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.032517 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.033371 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.033589 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.033679 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:34.036425 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:42753: BlockingRecv error: recv got EOF from 127.0.198.2:42753 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:36:34 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:34 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:34.231451 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:34.225909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:45215)
0929 18:36:34.226196 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:34.226210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:34.226311 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:34.226577 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:34.226580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:34.226767 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:34.226879 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:34.226884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.227636 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.227639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:34.228234 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:34.228243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.228718 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.228722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:34.228808 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:34.229402 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:34.229414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:34.231296 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
W20260929 18:36:34.231565 10016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:34.226058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:45215 (local address 127.0.198.62:37695)
0929 18:36:34.226178 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:34.226181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:34.226278 (+ 97us) server_negotiation.cc:407] Connection header received
0929 18:36:34.226378 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:34.226381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:34.226429 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:34.226504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:34.227008 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.227532 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.228362 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.228593 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.228672 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:34.231463 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:45215: BlockingRecv error: recv got EOF from 127.0.198.3:45215 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:36:34 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:34 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:34.851037 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:34.844925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:44605)
0929 18:36:34.845239 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:34.845257 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:34.845372 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:34.845703 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:34.845706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:34.845895 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:34.846042 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:34.846050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.846952 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.846955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:34.847593 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:34.847602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.848155 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.848158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:34.848245 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:34.848807 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:34.848822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:34.850920 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":72,"threads_started":1}
W20260929 18:36:34.851188 10020 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:34.845118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:44605 (local address 127.0.198.62:37695)
0929 18:36:34.845410 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:34.845414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:34.845427 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:36:34.845470 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:34.845472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:34.845514 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:34.845602 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:34.846253 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.846816 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.847737 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:34.848006 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:34.848108 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:34.851061 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:44605: BlockingRecv error: recv got EOF from 127.0.198.1:44605 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
Sep 29 18:36:35 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:35 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:35.042881 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:35.037374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:41237)
0929 18:36:35.037703 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:35.037719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:35.037843 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:35.038120 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:35.038123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:35.038313 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:35.038411 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:35.038417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.039198 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.039201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:35.039758 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:35.039766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.040189 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.040192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:35.040270 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.040833 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:35.040847 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:35.042730 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
W20260929 18:36:35.043017 10020 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:35.037514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:41237 (local address 127.0.198.62:37695)
0929 18:36:35.037703 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:35.037708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:35.037797 (+ 89us) server_negotiation.cc:407] Connection header received
0929 18:36:35.037936 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:35.037939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:35.038013 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:35.038079 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:35.038521 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.039089 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.039873 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.040090 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.040172 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.042920 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:41237: BlockingRecv error: recv got EOF from 127.0.198.2:41237 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 18:36:35 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:35 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:35.237303 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:35.232170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:40285)
0929 18:36:35.232451 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:35.232463 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:35.232555 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:35.232756 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:35.232758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:35.232943 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:35.233036 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:35.233041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.233743 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.233746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:35.234375 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:35.234385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.234833 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.234836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:35.234923 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.235485 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:35.235497 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:35.237189 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
W20260929 18:36:35.237474 10020 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:35.232266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:40285 (local address 127.0.198.62:37695)
0929 18:36:35.232384 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:35.232388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:35.232520 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:36:35.232623 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:35.232625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:35.232667 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:35.232733 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:35.233122 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.233647 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.234503 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.234726 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.234813 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.237359 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:40285: BlockingRecv error: recv got EOF from 127.0.198.3:40285 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 18:36:35 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:35 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:35.857596 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:35.851895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:59181)
0929 18:36:35.852182 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:35.852196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:35.852297 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:35.852528 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:35.852530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:35.852779 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:35.852876 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:35.852882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.853729 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.853732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:35.854398 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:35.854409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.854976 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.854979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:35.855046 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.855586 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:35.855599 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:35.857458 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
W20260929 18:36:35.857862 10024 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:35.852034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:59181 (local address 127.0.198.62:37695)
0929 18:36:35.852265 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:35.852268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:35.852284 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:36:35.852380 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:35.852383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:35.852425 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:35.852503 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:35.853025 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.853557 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.854539 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.854846 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.854974 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.857608 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:59181: BlockingRecv error: recv got EOF from 127.0.198.1:59181 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":56,"threads_started":1}
Sep 29 18:36:36 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:36 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:36.049357 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:36.043828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:51481)
0929 18:36:36.044132 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:36.044148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:36.044255 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:36.044567 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:36.044570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:36.044814 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:36.044920 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:36.044927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.045818 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.045822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:36.046472 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:36.046483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.046931 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.046935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:36.047017 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.047515 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:36.047528 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:36.049223 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
W20260929 18:36:36.049480 10024 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:36.044045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:51481 (local address 127.0.198.62:37695)
0929 18:36:36.044174 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:36.044179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:36.044197 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:36:36.044376 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:36.044379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:36.044432 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:36.044507 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:36.045052 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.045714 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.046589 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.046826 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.046915 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.049364 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:51481: BlockingRecv error: recv got EOF from 127.0.198.2:51481 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 18:36:36 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:36 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:36.243927 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:36.238228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:37547)
0929 18:36:36.238508 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:36.238523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:36.238718 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:36.238972 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:36.238975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:36.239183 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:36.239276 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:36.239283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.240022 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.240025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:36.240581 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:36.240588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.241031 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.241034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:36.241113 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.241596 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:36.241609 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:36.243780 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
W20260929 18:36:36.243997 10024 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:36.238354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:37547 (local address 127.0.198.62:37695)
0929 18:36:36.238509 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:36.238513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:36.238583 (+ 70us) server_negotiation.cc:407] Connection header received
0929 18:36:36.238821 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:36.238824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:36.238875 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:36.238939 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:36.239393 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.239920 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.240702 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.240937 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.241020 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.243898 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:37547: BlockingRecv error: recv got EOF from 127.0.198.3:37547 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 18:36:36 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:36 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:36.871434 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:36.858399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:40903)
0929 18:36:36.858717 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:36.858735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:36.858844 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:36.859072 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:36.859074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:36.859264 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:36.859367 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:36.859372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.860171 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.860178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:36.860820 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:36.860830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.861411 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.861414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:36.861509 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.869290 (+ 7781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:36.869305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:36.871302 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
W20260929 18:36:36.871655 10028 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:36.858526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:40903 (local address 127.0.198.62:37695)
0929 18:36:36.858848 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:36.858852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:36.858866 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:36.858905 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:36.858908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:36.858948 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:36.859026 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:36.859472 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.860049 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.860939 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.861273 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.861375 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.871502 (+ 10127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:40903: BlockingRecv error: recv got EOF from 127.0.198.1:40903 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":82,"threads_started":1}
Sep 29 18:36:37 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:37 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:37.055125 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:37.050085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:54847)
0929 18:36:37.050353 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:37.050366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:37.050469 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:37.050728 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:37.050731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:37.050912 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:37.051005 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:37.051011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.051714 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.051717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:37.052305 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:37.052313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.052746 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.052749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:37.052813 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:37.053307 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:37.053319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:37.055011 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
W20260929 18:36:37.055292 10028 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:37.050119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:54847 (local address 127.0.198.62:37695)
0929 18:36:37.050262 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:37.050267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:37.050440 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:36:37.050558 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:37.050561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:37.050615 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:37.050693 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:37.051097 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.051614 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.052416 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.052643 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.052728 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:37.055167 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:54847: BlockingRecv error: recv got EOF from 127.0.198.2:54847 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:36:37 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:37 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.3@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:37.250365 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:37.244897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.3:45443)
0929 18:36:37.245170 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:37.245183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:37.245281 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:37.245525 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:37.245527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:37.245788 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:37.245890 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:37.245896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.246738 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.246741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:37.247323 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:37.247331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.247844 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.247848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:37.247933 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:37.248446 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:37.248460 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:37.250244 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
W20260929 18:36:37.250483 10028 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:37.245002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:45443 (local address 127.0.198.62:37695)
0929 18:36:37.245129 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:37.245133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:37.245248 (+ 115us) server_negotiation.cc:407] Connection header received
0929 18:36:37.245379 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:37.245381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:37.245427 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:37.245484 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:37.246015 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.246561 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.247450 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.247715 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.247796 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:37.250380 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.3:45443: BlockingRecv error: recv got EOF from 127.0.198.3:45443 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 18:36:37 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:37 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.1@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:37.877424 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:37.872304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.1:54285)
0929 18:36:37.872579 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:37.872592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:37.872684 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:37.872934 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:37.872937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:37.873127 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:37.873220 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:37.873225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.873961 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.873964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:37.874585 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:37.874594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.874978 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.874980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:37.875035 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:37.875498 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:37.875509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:37.877310 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
W20260929 18:36:37.877561 10032 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:37.872375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:54285 (local address 127.0.198.62:37695)
0929 18:36:37.872599 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:37.872602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:37.872659 (+ 57us) server_negotiation.cc:407] Connection header received
0929 18:36:37.872757 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:37.872758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:37.872798 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:37.872869 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:37.873317 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.873858 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.874682 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:37.874895 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:37.874969 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:37.877442 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.1:54285: BlockingRecv error: recv got EOF from 127.0.198.1:54285 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
Sep 29 18:36:38 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:36:38 dist-test-slave-q5qx krb5kdc[8974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.198.2@KRBTEST.COM for oryx/127.0.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:36:38.061494 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.056145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:37695 (local address 127.0.198.2:58673)
0929 18:36:38.056411 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:38.056424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:38.056523 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:38.056769 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:38.056771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:38.056944 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:38.057038 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.057043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.057950 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.057953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.058647 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.058658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.059144 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.059148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.059241 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.059719 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:38.059731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:38.061377 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.198.62:37695: Server oryx/127.0.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
W20260929 18:36:38.061643 10032 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:38.056280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:58673 (local address 127.0.198.62:37695)
0929 18:36:38.056435 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.056438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.056489 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:36:38.056609 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.056613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.056663 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.056743 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.057185 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.057816 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.058767 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.059028 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.059127 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.061532 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.198.2:58673: BlockingRecv error: recv got EOF from 127.0.198.2:58673 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 18:36:38.150621 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 9557
I20260929 18:36:38.155465 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 9709
I20260929 18:36:38.160569 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 9841
I20260929 18:36:38.165095 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 9487
2026-09-29T18:36:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17927 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T18:36:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:38Z Disabled control of system clock
I20260929 18:36:38.192884 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:46461
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:46461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:36:38.302561 10037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:38.302834 10037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:38.302866 10037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:38.306910 10037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:38.306977 10037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:38.306991 10037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:38.307006 10037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:38.307021 10037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:38.311775 10037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:46461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:46461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:38.312855 10037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:38.314275 10037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:38.322921 10045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:38.322968 10042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:38.322937 10043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.322989 10037 server_base.cc:1034] running on GCE node
I20260929 18:36:38.323769 10037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:38.324836 10037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:38.326012 10037 hybrid_clock.cc:648] HybridClock initialized: now 1790706998325994 us; error 31 us; skew 500 ppm
I20260929 18:36:38.327800 10037 webserver.cc:460] Webserver started at http://127.0.198.62:44329/ using document root <none> and password file <none>
I20260929 18:36:38.328383 10037 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:38.328449 10037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:38.328639 10037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:38.330698 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "a3302600f3fa4185ad78a63748e1442b"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "76c9651ff7e33b934706b21f833dfa7e"
server_key_iv: "2f774e7e7cd791620da3fcc38aba1f62"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.331262 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "a3302600f3fa4185ad78a63748e1442b"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "76c9651ff7e33b934706b21f833dfa7e"
server_key_iv: "2f774e7e7cd791620da3fcc38aba1f62"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.334959 10037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:36:38.337325 10051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.338624 10037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:38.338737 10037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "a3302600f3fa4185ad78a63748e1442b"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "76c9651ff7e33b934706b21f833dfa7e"
server_key_iv: "2f774e7e7cd791620da3fcc38aba1f62"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.338901 10037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:38.348029 10037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:38.348652 10037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:38.348810 10037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:38.355384 10037 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:46461
I20260929 18:36:38.355399 10103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:46461 every 8 connection(s)
I20260929 18:36:38.356487 10037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:38.357947 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10037
I20260929 18:36:38.358058 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:38.358325 792 external_mini_cluster.cc:1428] Setting key 5ce34f35ddc911b96d2c9835a917d054
I20260929 18:36:38.359724 10104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:38.365016 10104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b: Bootstrap starting.
I20260929 18:36:38.367301 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.359569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.0.198.62:46461)
0929 18:36:38.360297 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.360317 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.360356 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:36:38.360986 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.361001 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.361392 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.361694 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.362144 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.363109 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.363950 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.364414 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.364553 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.365338 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:38.365343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:38.365355 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:38.365746 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:38.365800 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:38.365821 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:38.365901 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:38.366057 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:36:38.366388 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":155,"threads_started":1}
I20260929 18:36:38.367372 10104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:38.368206 10104 log.cc:824] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:38.370146 10104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b: No bootstrap required, opened a new log
I20260929 18:36:38.373386 10104 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3302600f3fa4185ad78a63748e1442b" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46461 } }
I20260929 18:36:38.373598 10104 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:38.373633 10104 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3302600f3fa4185ad78a63748e1442b, State: Initialized, Role: FOLLOWER
I20260929 18:36:38.374166 10104 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3302600f3fa4185ad78a63748e1442b" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46461 } }
I20260929 18:36:38.374322 10104 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:38.374409 10104 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:38.374505 10104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:38.375383 10104 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3302600f3fa4185ad78a63748e1442b" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46461 } }
I20260929 18:36:38.375684 10104 leader_election.cc:302] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3302600f3fa4185ad78a63748e1442b; no voters:
I20260929 18:36:38.375952 10104 leader_election.cc:288] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:38.376042 10109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:38.376260 10109 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [term 1 LEADER]: Becoming Leader. State: Replica: a3302600f3fa4185ad78a63748e1442b, State: Running, Role: LEADER
I20260929 18:36:38.376554 10109 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3302600f3fa4185ad78a63748e1442b" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46461 } }
I20260929 18:36:38.377022 10104 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:38.378183 10111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3302600f3fa4185ad78a63748e1442b. Latest consensus state: current_term: 1 leader_uuid: "a3302600f3fa4185ad78a63748e1442b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3302600f3fa4185ad78a63748e1442b" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46461 } } }
I20260929 18:36:38.378167 10110 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3302600f3fa4185ad78a63748e1442b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3302600f3fa4185ad78a63748e1442b" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 46461 } } }
I20260929 18:36:38.378384 10110 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:38.378628 10111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:38.381264 10124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:38.381326 10124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:38.381399 10119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:38.382097 10119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:38.387391 10119 catalog_manager.cc:1347] Generated new cluster ID: 226029376f644631b3ca7a35dc50b549
I20260929 18:36:38.387462 10119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:38.406076 10119 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:38.407040 10119 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:38.415303 10119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3302600f3fa4185ad78a63748e1442b: Generated new TSK 0
I20260929 18:36:38.416100 10119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:38.428856 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46461
--builtin_ntp_servers=127.0.198.20:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:36:38.538100 10128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:38.538370 10128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:38.538406 10128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:38.542407 10128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:38.542461 10128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:38.542515 10128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:38.547292 10128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:38.548342 10128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:38.549672 10128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:38.559165 10134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:38.559165 10136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.559269 10128 server_base.cc:1034] running on GCE node
W20260929 18:36:38.559289 10133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.559911 10128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:38.560514 10128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:38.561685 10128 hybrid_clock.cc:648] HybridClock initialized: now 1790706998561665 us; error 33 us; skew 500 ppm
I20260929 18:36:38.563512 10128 webserver.cc:460] Webserver started at http://127.0.198.1:46657/ using document root <none> and password file <none>
I20260929 18:36:38.564169 10128 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:38.564234 10128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:38.564421 10128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:38.566498 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "a33eac4223b54a28a0215c45271caddb"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "4628b4ac43131602cddc2264b1c5b1ca"
server_key_iv: "48bc23d72da7da0fca282f214d21e4a2"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.567031 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "a33eac4223b54a28a0215c45271caddb"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "4628b4ac43131602cddc2264b1c5b1ca"
server_key_iv: "48bc23d72da7da0fca282f214d21e4a2"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.570820 10128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:38.573365 10142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.574745 10128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:38.574847 10128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "a33eac4223b54a28a0215c45271caddb"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "4628b4ac43131602cddc2264b1c5b1ca"
server_key_iv: "48bc23d72da7da0fca282f214d21e4a2"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.574937 10128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:38.591888 10128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:38.592558 10128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:38.592705 10128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:38.593240 10128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:38.594151 10128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:38.594199 10128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.594246 10128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:38.594265 10128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.602764 10128 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:38539
I20260929 18:36:38.602785 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:38539 every 8 connection(s)
I20260929 18:36:38.603621 10128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:38.604624 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10128
I20260929 18:36:38.604743 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:38.604987 792 external_mini_cluster.cc:1428] Setting key 6c029e8669393c28e7f6084e9bef9be0
I20260929 18:36:38.606966 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46461
--builtin_ntp_servers=127.0.198.20:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:36:38.613911 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.605343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:42621 (local address 127.0.198.62:46461)
0929 18:36:38.605460 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.605463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.606838 (+ 1375us) server_negotiation.cc:407] Connection header received
0929 18:36:38.608540 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.608543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.608600 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.608756 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.609564 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.610362 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.611660 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.611999 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.612109 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.613378 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:38.613383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:38.613386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:38.613402 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:38.613425 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:38.613436 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:38.613514 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:38.613682 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:36:38.613757 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:36:38.614841 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.605846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:46461 (local address 127.0.198.1:42621)
0929 18:36:38.606655 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:38.606707 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:38.608302 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:38.608863 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:38.608870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:38.608953 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:38.609431 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.609443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.610529 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.610532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.611460 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.611470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.612154 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.612157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.612303 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.613068 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:38.613070 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:38.613126 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:38.613208 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:38.613518 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:38.613524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:38.613529 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:38.613658 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:36:38.613931 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
I20260929 18:36:38.616032 10256 heartbeater.cc:347] Connected to a master server at 127.0.198.62:46461
I20260929 18:36:38.616285 10256 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:38.616914 10256 heartbeater.cc:511] Master 127.0.198.62:46461 requested a full tablet report, sending...
I20260929 18:36:38.618512 10068 ts_manager.cc:194] Registered new tserver with Master: a33eac4223b54a28a0215c45271caddb (127.0.198.1:38539)
I20260929 18:36:38.620016 10068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.1:42621
W20260929 18:36:38.716877 10259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:38.717129 10259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:38.717163 10259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:38.721216 10259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:38.721271 10259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:38.721324 10259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:38.726050 10259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:38.727027 10259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:38.728353 10259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:38.737615 10264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.737722 10259 server_base.cc:1034] running on GCE node
W20260929 18:36:38.737620 10265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:38.737617 10267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.738220 10259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:38.738821 10259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:38.739971 10259 hybrid_clock.cc:648] HybridClock initialized: now 1790706998739952 us; error 32 us; skew 500 ppm
I20260929 18:36:38.742136 10259 webserver.cc:460] Webserver started at http://127.0.198.2:37951/ using document root <none> and password file <none>
I20260929 18:36:38.742873 10259 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:38.742945 10259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:38.743158 10259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:38.745129 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "9720ee95c0634eeab26b279a8be8794f"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "def049891cb1772edd39d6dd0aa3fcfd"
server_key_iv: "845f75b839718a9c0cb826f514344590"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.745667 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "9720ee95c0634eeab26b279a8be8794f"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "def049891cb1772edd39d6dd0aa3fcfd"
server_key_iv: "845f75b839718a9c0cb826f514344590"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.749203 10259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 18:36:38.751688 10273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.752956 10259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:38.753064 10259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "9720ee95c0634eeab26b279a8be8794f"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "def049891cb1772edd39d6dd0aa3fcfd"
server_key_iv: "845f75b839718a9c0cb826f514344590"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.753168 10259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:38.771597 10259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:38.772290 10259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:38.772466 10259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:38.773087 10259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:38.774015 10259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:38.774065 10259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.774116 10259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:38.774142 10259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.782932 10259 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:37737
I20260929 18:36:38.783030 10386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:37737 every 8 connection(s)
I20260929 18:36:38.783836 10259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:38.793118 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.785537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:39847 (local address 127.0.198.62:46461)
0929 18:36:38.785689 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.785693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.786798 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 18:36:38.788123 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.788126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.788171 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.788245 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.789382 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.789959 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.791371 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.791589 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.791676 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.792665 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:38.792668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:38.792670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:38.792682 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:38.792723 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:38.792731 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:38.792785 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:38.792947 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:36:38.793000 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:36:38.793470 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10259
I20260929 18:36:38.793592 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:38.793896 792 external_mini_cluster.cc:1428] Setting key f4da63a3369b5d04f713fcf72089d6d7
I20260929 18:36:38.794302 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.786000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:46461 (local address 127.0.198.2:39847)
0929 18:36:38.786657 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:38.786697 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:38.787931 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:38.788501 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:38.788586 (+ 85us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:38.788684 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:38.789215 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.789229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.790144 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.790148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.791239 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.791249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.791722 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.791725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.791867 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.792414 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:38.792416 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:38.792457 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:38.792524 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:38.792819 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:38.792826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:38.792831 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:38.792945 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:36:38.793181 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260929 18:36:38.795485 10387 heartbeater.cc:347] Connected to a master server at 127.0.198.62:46461
I20260929 18:36:38.795740 10387 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:38.795802 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46461
--builtin_ntp_servers=127.0.198.20:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:36:38.796332 10387 heartbeater.cc:511] Master 127.0.198.62:46461 requested a full tablet report, sending...
I20260929 18:36:38.797417 10067 ts_manager.cc:194] Registered new tserver with Master: 9720ee95c0634eeab26b279a8be8794f (127.0.198.2:37737)
I20260929 18:36:38.798169 10067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.2:39847
W20260929 18:36:38.904410 10390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:38.904668 10390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:38.904704 10390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:38.908696 10390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:38.908748 10390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:38.908799 10390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:38.913373 10390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:38.914412 10390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:38.915846 10390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:38.925040 10395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:38.925060 10398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:38.925065 10396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.925362 10390 server_base.cc:1034] running on GCE node
I20260929 18:36:38.925772 10390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:38.926420 10390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:38.927577 10390 hybrid_clock.cc:648] HybridClock initialized: now 1790706998927559 us; error 28 us; skew 500 ppm
I20260929 18:36:38.929399 10390 webserver.cc:460] Webserver started at http://127.0.198.3:36653/ using document root <none> and password file <none>
I20260929 18:36:38.929949 10390 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:38.930058 10390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:38.930277 10390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:38.932431 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "fec7fa92aa1e4558943cf14038d4cc7f"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "2f19c769b286103f7544c5a41e2b800f"
server_key_iv: "a1202941fe6da65c017cf0cda084b7c5"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.932981 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "fec7fa92aa1e4558943cf14038d4cc7f"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "2f19c769b286103f7544c5a41e2b800f"
server_key_iv: "a1202941fe6da65c017cf0cda084b7c5"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.937011 10390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:38.939627 10404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.940902 10390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:38.941009 10390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "fec7fa92aa1e4558943cf14038d4cc7f"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-q5qx"
server_key: "2f19c769b286103f7544c5a41e2b800f"
server_key_iv: "a1202941fe6da65c017cf0cda084b7c5"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.941110 10390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:38.969741 10390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:38.970520 10390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:38.970706 10390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:38.971290 10390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:38.972260 10390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:38.972308 10390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.972362 10390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:38.972389 10390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.980646 10390 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:35337
I20260929 18:36:38.980661 10517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:35337 every 8 connection(s)
I20260929 18:36:38.981549 10390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:38.982311 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10390
I20260929 18:36:38.982406 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:38.982656 792 external_mini_cluster.cc:1428] Setting key 0533ed4398ac3a155f6eef8e3401aa25
I20260929 18:36:38.990854 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.983371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:53037 (local address 127.0.198.62:46461)
0929 18:36:38.983499 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.983503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.984782 (+ 1279us) server_negotiation.cc:407] Connection header received
0929 18:36:38.986258 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.986261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.986314 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.986385 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.987278 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.987801 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.988944 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.989171 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.989257 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.990380 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:38.990383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:38.990386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:38.990400 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:38.990431 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:38.990439 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:38.990498 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:38.990638 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:36:38.990702 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:36:38.991786 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.983898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:46461 (local address 127.0.198.3:53037)
0929 18:36:38.984647 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:38.984704 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:38.986058 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:38.986559 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:38.986566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:38.986655 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:38.987146 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.987160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.987919 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.987922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.988828 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:38.988837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:38.989316 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:38.989320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:38.989488 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:38.990115 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:38.990117 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:38.990161 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:38.990231 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:38.990506 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:38.990515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:38.990520 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:38.990635 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:36:38.990931 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260929 18:36:38.993243 10518 heartbeater.cc:347] Connected to a master server at 127.0.198.62:46461
I20260929 18:36:38.993462 10518 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:38.993997 10518 heartbeater.cc:511] Master 127.0.198.62:46461 requested a full tablet report, sending...
I20260929 18:36:38.995036 10067 ts_manager.cc:194] Registered new tserver with Master: fec7fa92aa1e4558943cf14038d4cc7f (127.0.198.3:35337)
I20260929 18:36:38.995702 10067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.3:53037
I20260929 18:36:38.996408 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:36:38.998967 10107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:38.998223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.0.198.62:46461)
0929 18:36:38.998338 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.998342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.998493 (+ 151us) server_negotiation.cc:407] Connection header received
0929 18:36:38.998588 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.998590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.998624 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.998685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.998873 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39298: BlockingRecv error: recv got EOF from 127.0.0.1:39298 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:36:38.999195 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10128
I20260929 18:36:39.003707 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10259
I20260929 18:36:39.008261 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10390
I20260929 18:36:39.012990 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10037
2026-09-29T18:36:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (847 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T18:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:39Z Disabled control of system clock
I20260929 18:36:39.040004 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:40147
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:40147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:36:39.149680 10529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:39.149963 10529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:39.150029 10529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:39.154093 10529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:39.154161 10529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:39.154175 10529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:39.154192 10529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:39.154223 10529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:39.159138 10529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:40147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:40147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:39.160238 10529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:39.161654 10529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:39.170300 10534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:39.170377 10529 server_base.cc:1034] running on GCE node
W20260929 18:36:39.170305 10535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.170305 10537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:39.171126 10529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:39.172194 10529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:39.173352 10529 hybrid_clock.cc:648] HybridClock initialized: now 1790706999173333 us; error 30 us; skew 500 ppm
I20260929 18:36:39.175153 10529 webserver.cc:460] Webserver started at http://127.0.198.62:34693/ using document root <none> and password file <none>
I20260929 18:36:39.175740 10529 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:39.175805 10529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:39.175990 10529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:39.178033 10529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "c3d30177f66e6a8582cc4db9d84e61d0"
server_key_iv: "458322438effc823e5c34445f2ef3252"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.178581 10529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "c3d30177f66e6a8582cc4db9d84e61d0"
server_key_iv: "458322438effc823e5c34445f2ef3252"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.182335 10529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:36:39.184763 10543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.186151 10529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:39.186270 10529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "c3d30177f66e6a8582cc4db9d84e61d0"
server_key_iv: "458322438effc823e5c34445f2ef3252"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.186376 10529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:39.190958 10529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:39.191553 10529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:39.191705 10529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:39.197945 10529 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:40147
I20260929 18:36:39.197960 10595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:40147 every 8 connection(s)
I20260929 18:36:39.198869 10529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:39.202042 10596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:39.205403 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10529
I20260929 18:36:39.205528 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:39.205832 792 external_mini_cluster.cc:1428] Setting key e9f92b5ddc4440afa8e66793f2644bfa
I20260929 18:36:39.207965 10596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace: Bootstrap starting.
I20260929 18:36:39.210201 10596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:39.210983 10596 log.cc:824] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:39.211552 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.207427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.198.62:40147)
0929 18:36:39.208059 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:39.208079 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:39.208117 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:36:39.208803 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:39.208819 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:39.209164 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:39.209476 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:39.210266 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:39.210277 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:39.210293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:39.210305 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:39.210366 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:39.210383 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:39.210480 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:39.210609 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:36:39.210844 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260929 18:36:39.212802 10596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace: No bootstrap required, opened a new log
I20260929 18:36:39.216359 10596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40147 } }
I20260929 18:36:39.216615 10596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:39.216678 10596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838b2ae2a9f5496ba2fbb1c54e7c6ace, State: Initialized, Role: FOLLOWER
I20260929 18:36:39.217057 10596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40147 } }
I20260929 18:36:39.217171 10596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:39.217227 10596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:39.217310 10596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:39.218185 10596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40147 } }
I20260929 18:36:39.218483 10596 leader_election.cc:302] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838b2ae2a9f5496ba2fbb1c54e7c6ace; no voters:
I20260929 18:36:39.218766 10596 leader_election.cc:288] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:39.218858 10601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:39.219058 10601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [term 1 LEADER]: Becoming Leader. State: Replica: 838b2ae2a9f5496ba2fbb1c54e7c6ace, State: Running, Role: LEADER
I20260929 18:36:39.219331 10601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40147 } }
I20260929 18:36:39.219856 10596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:39.220749 10602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40147 } } }
I20260929 18:36:39.220863 10602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:39.221120 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838b2ae2a9f5496ba2fbb1c54e7c6ace. Latest consensus state: current_term: 1 leader_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838b2ae2a9f5496ba2fbb1c54e7c6ace" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 40147 } } }
I20260929 18:36:39.221191 10610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:39.221246 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:39.224225 10610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:39.229660 10610 catalog_manager.cc:1347] Generated new cluster ID: 3a486b7b26184728a3cef4f029408f6b
I20260929 18:36:39.229738 10610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:39.247062 10610 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:39.247396 10610 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:39.256964 10610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838b2ae2a9f5496ba2fbb1c54e7c6ace: Generated new TSK 0
I20260929 18:36:39.257655 10610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:39.262224 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:40147
--builtin_ntp_servers=127.0.198.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:36:39.371860 10620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:39.372135 10620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:39.372190 10620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:39.376219 10620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:39.376279 10620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:39.376355 10620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:39.381228 10620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:40147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:39.382691 10620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:39.384513 10620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:39.395926 10626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.396242 10625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.396430 10628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:39.398111 10620 server_base.cc:1034] running on GCE node
I20260929 18:36:39.398635 10620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:39.399435 10620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:39.400616 10620 hybrid_clock.cc:648] HybridClock initialized: now 1790706999400543 us; error 98 us; skew 500 ppm
I20260929 18:36:39.403002 10620 webserver.cc:460] Webserver started at http://127.0.198.1:40967/ using document root <none> and password file <none>
I20260929 18:36:39.403796 10620 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:39.403891 10620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:39.404130 10620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:39.407117 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "3f62fc80ae02488c9842b6c26af33afd"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "b7a8f66a556200e4891400cf0b92dcb9"
server_key_iv: "5b5ca572881ca974eee8a11b50e60f91"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.407821 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "3f62fc80ae02488c9842b6c26af33afd"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "b7a8f66a556200e4891400cf0b92dcb9"
server_key_iv: "5b5ca572881ca974eee8a11b50e60f91"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.412850 10620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:36:39.415877 10634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.417413 10620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:39.417543 10620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "3f62fc80ae02488c9842b6c26af33afd"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "b7a8f66a556200e4891400cf0b92dcb9"
server_key_iv: "5b5ca572881ca974eee8a11b50e60f91"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.417666 10620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:39.424300 10620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:39.425069 10620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:39.425258 10620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:39.425961 10620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:39.427173 10620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:39.427232 10620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.427285 10620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:39.427315 10620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.438347 10620 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:36941
I20260929 18:36:39.438381 10747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:36941 every 8 connection(s)
I20260929 18:36:39.439249 10620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:39.445267 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.441035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:58731 (local address 127.0.198.62:40147)
0929 18:36:39.441195 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:39.441200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:39.442334 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 18:36:39.443607 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:39.443610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:39.443648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:39.443723 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:39.444880 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:39.444883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:39.444885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:39.444897 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:39.444917 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:39.444924 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:39.445008 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:39.445098 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:36:39.445131 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:36:39.446273 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.441485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:40147 (local address 127.0.198.1:58731)
0929 18:36:39.442196 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:39.442236 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:39.443405 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:39.443868 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:39.443874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:39.443963 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:39.444500 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:39.444501 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:39.444541 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:39.444673 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:39.444997 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:39.445002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:39.445006 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:36:39.445130 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:36:39.445335 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20260929 18:36:39.447466 10748 heartbeater.cc:347] Connected to a master server at 127.0.198.62:40147
I20260929 18:36:39.447690 10748 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:39.448086 10748 heartbeater.cc:511] Master 127.0.198.62:40147 requested a full tablet report, sending...
I20260929 18:36:39.448360 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10620
I20260929 18:36:39.448455 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:39.448756 792 external_mini_cluster.cc:1428] Setting key 9d82dc407f482acea33e2ae521b8f693
I20260929 18:36:39.449761 10555 ts_manager.cc:194] Registered new tserver with Master: 3f62fc80ae02488c9842b6c26af33afd (127.0.198.1:36941)
I20260929 18:36:39.451033 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:40147
--builtin_ntp_servers=127.0.198.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:36:39.559342 10751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:39.559612 10751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:39.559650 10751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:39.563735 10751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:39.563802 10751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:39.563859 10751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:39.568665 10751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:40147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:39.569722 10751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:39.571127 10751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:39.580592 10759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:39.580660 10751 server_base.cc:1034] running on GCE node
W20260929 18:36:39.580650 10756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.580608 10757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:39.581286 10751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:39.582091 10751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:39.583273 10751 hybrid_clock.cc:648] HybridClock initialized: now 1790706999583243 us; error 43 us; skew 500 ppm
I20260929 18:36:39.585189 10751 webserver.cc:460] Webserver started at http://127.0.198.2:35991/ using document root <none> and password file <none>
I20260929 18:36:39.585757 10751 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:39.585817 10751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:39.586043 10751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:39.588088 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "56b999a5110646e6bb8ef8c4ed0dfed4"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "66935c33ada5c53b7ce06e8bcb390a32"
server_key_iv: "238ad7d08d13d348468dd8de81f8b167"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.588615 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "56b999a5110646e6bb8ef8c4ed0dfed4"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "66935c33ada5c53b7ce06e8bcb390a32"
server_key_iv: "238ad7d08d13d348468dd8de81f8b167"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.592561 10751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:36:39.594966 10765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.596222 10751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:39.596329 10751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "56b999a5110646e6bb8ef8c4ed0dfed4"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "66935c33ada5c53b7ce06e8bcb390a32"
server_key_iv: "238ad7d08d13d348468dd8de81f8b167"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.596423 10751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:39.600979 10751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:39.601552 10751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:39.601689 10751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:39.602216 10751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:39.603075 10751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:39.603119 10751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.603158 10751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:39.603185 10751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.611424 10751 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:37661
I20260929 18:36:39.611445 10878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:37661 every 8 connection(s)
I20260929 18:36:39.612299 10751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:39.617105 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10751
I20260929 18:36:39.617219 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:39.617465 792 external_mini_cluster.cc:1428] Setting key 4cb97619878fef1156ca44a1e1132018
I20260929 18:36:39.618367 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.614088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:44069 (local address 127.0.198.62:40147)
0929 18:36:39.614231 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:39.614234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:39.615285 (+ 1051us) server_negotiation.cc:407] Connection header received
0929 18:36:39.616639 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:39.616642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:39.616680 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:39.616743 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:39.617913 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:39.617916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:39.617918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:39.617929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:39.617948 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:39.617955 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:39.618055 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:39.618206 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 18:36:39.618249 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:36:39.619285 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:40147
--builtin_ntp_servers=127.0.198.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:36:39.619307 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.614499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:40147 (local address 127.0.198.2:44069)
0929 18:36:39.615146 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:39.615186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:39.616421 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:39.616923 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:39.616929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:39.617016 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:39.617645 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:39.617646 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:39.617692 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:39.617760 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:39.618048 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:39.618056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:39.618069 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 18:36:39.618237 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 18:36:39.618427 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260929 18:36:39.620321 10879 heartbeater.cc:347] Connected to a master server at 127.0.198.62:40147
I20260929 18:36:39.620680 10879 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:39.621057 10879 heartbeater.cc:511] Master 127.0.198.62:40147 requested a full tablet report, sending...
I20260929 18:36:39.622051 10555 ts_manager.cc:194] Registered new tserver with Master: 56b999a5110646e6bb8ef8c4ed0dfed4 (127.0.198.2:37661)
W20260929 18:36:39.728039 10882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:39.728371 10882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:39.728440 10882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:39.732559 10882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:39.732614 10882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:39.732669 10882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:39.737484 10882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:40147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:39.738536 10882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:39.739848 10882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:39.749766 10888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.749742 10887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.749744 10890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:39.749894 10882 server_base.cc:1034] running on GCE node
I20260929 18:36:39.750452 10882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:39.751063 10882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:39.752242 10882 hybrid_clock.cc:648] HybridClock initialized: now 1790706999752209 us; error 41 us; skew 500 ppm
I20260929 18:36:39.754032 10882 webserver.cc:460] Webserver started at http://127.0.198.3:37195/ using document root <none> and password file <none>
I20260929 18:36:39.754585 10882 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:39.754648 10882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:39.754843 10882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:39.756857 10882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "d8068436bb1d4124b8520273992b39b2"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "3a688d699a6c5730523c606667615506"
server_key_iv: "4ad9fcddfb428bf63730b0c8b8c94f43"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.757424 10882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "d8068436bb1d4124b8520273992b39b2"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "3a688d699a6c5730523c606667615506"
server_key_iv: "4ad9fcddfb428bf63730b0c8b8c94f43"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.761061 10882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:36:39.763368 10896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.764603 10882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:36:39.764714 10882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "d8068436bb1d4124b8520273992b39b2"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-q5qx"
server_key: "3a688d699a6c5730523c606667615506"
server_key_iv: "4ad9fcddfb428bf63730b0c8b8c94f43"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.764810 10882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:39.769129 10882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:39.769680 10882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:39.769822 10882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:39.770372 10882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:39.771214 10882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:39.771257 10882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.771302 10882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:39.771329 10882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.779224 10882 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:34931
I20260929 18:36:39.779271 11009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:34931 every 8 connection(s)
I20260929 18:36:39.780035 10882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:39.785413 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 10882
I20260929 18:36:39.785532 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:39.785825 792 external_mini_cluster.cc:1428] Setting key 1042a743b0467d1a78164a4c4d4b7f2c
I20260929 18:36:39.786087 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.781746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:49355 (local address 127.0.198.62:40147)
0929 18:36:39.781891 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:39.781895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:39.783105 (+ 1210us) server_negotiation.cc:407] Connection header received
0929 18:36:39.784355 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:39.784358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:39.784393 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:39.784467 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:39.785602 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:39.785607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:39.785610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:39.785624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:39.785647 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:39.785659 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:39.785737 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:39.785904 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:36:39.785949 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:36:39.787283 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.782214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:40147 (local address 127.0.198.3:49355)
0929 18:36:39.782966 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:39.783005 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:39.784169 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:39.784647 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:39.784654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:39.784748 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:39.785267 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:39.785269 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:39.785322 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:39.785411 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:39.785753 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:39.785762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:39.785768 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:36:39.785941 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 18:36:39.786368 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":103,"threads_started":1}
I20260929 18:36:39.788304 11010 heartbeater.cc:347] Connected to a master server at 127.0.198.62:40147
I20260929 18:36:39.788484 11010 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:39.788841 11010 heartbeater.cc:511] Master 127.0.198.62:40147 requested a full tablet report, sending...
I20260929 18:36:39.789784 10555 ts_manager.cc:194] Registered new tserver with Master: d8068436bb1d4124b8520273992b39b2 (127.0.198.3:34931)
I20260929 18:36:39.799796 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:36:39.802554 10599 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:39.801707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.0.198.62:40147)
0929 18:36:39.801806 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:39.801809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:39.802032 (+ 223us) server_negotiation.cc:407] Connection header received
0929 18:36:39.802160 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:39.802162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:39.802196 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:39.802253 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:39.802447 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47340: BlockingRecv error: recv got EOF from 127.0.0.1:47340 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:36:39.802870 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10620
I20260929 18:36:39.807740 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10751
I20260929 18:36:39.812086 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10882
I20260929 18:36:39.816829 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 10529
2026-09-29T18:36:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (803 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:39 dist-test-slave-q5qx krb5kdc[11020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:39 dist-test-slave-q5qx krb5kdc[11020](info): set up 2 sockets
Sep 29 18:36:39 dist-test-slave-q5qx krb5kdc[11020](info): commencing operation
krb5kdc: starting...
W20260929 18:36:40.778666 792 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:40 dist-test-slave-q5qx krb5kdc[11020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:36:40.901034 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42329
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:41.009008 11033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:41.009289 11033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:41.009342 11033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:41.013655 11033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:41.013737 11033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:41.013767 11033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:41.013787 11033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:41.013808 11033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:41.019748 11033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:41.020815 11033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:41.021799 11033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:41.030229 11041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.030272 11038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.030268 11039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:41.030427 11033 server_base.cc:1034] running on GCE node
I20260929 18:36:41.031072 11033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:41.032205 11033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:41.033368 11033 hybrid_clock.cc:648] HybridClock initialized: now 1790707001033353 us; error 28 us; skew 500 ppm
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:41.036120 11033 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:36:41.037055 11033 webserver.cc:460] Webserver started at http://127.0.198.62:44647/ using document root <none> and password file <none>
I20260929 18:36:41.037631 11033 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:41.037703 11033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:41.037887 11033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:41.039929 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "70893d2ec9754021bf3b2b9af0187469"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "fc906fbc8440576e7492d11022f670ca"
server_key_iv: "3c6e5f7a843549173419c7fb33cdb264"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.040467 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "70893d2ec9754021bf3b2b9af0187469"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "fc906fbc8440576e7492d11022f670ca"
server_key_iv: "3c6e5f7a843549173419c7fb33cdb264"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.044130 11033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:36:41.046499 11048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.047700 11033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:41.047798 11033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "70893d2ec9754021bf3b2b9af0187469"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "fc906fbc8440576e7492d11022f670ca"
server_key_iv: "3c6e5f7a843549173419c7fb33cdb264"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.047883 11033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:41.069486 11033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:41.072569 11033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:41.072729 11033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:41.079500 11033 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:42329
I20260929 18:36:41.079768 11100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:42329 every 8 connection(s)
I20260929 18:36:41.080449 11033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:41.083391 11101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:41.087137 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 11033
I20260929 18:36:41.087266 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:36:41.087560 792 external_mini_cluster.cc:1428] Setting key d6ba4596ae6a7d445eb8fb3a08dc5ae0
I20260929 18:36:41.089694 11101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469: Bootstrap starting.
I20260929 18:36:41.092224 11101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:41.092963 11101 log.cc:824] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:41.095156 11101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469: No bootstrap required, opened a new log
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:41.098659 11101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70893d2ec9754021bf3b2b9af0187469" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:36:41.098930 11101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:41.098994 11101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70893d2ec9754021bf3b2b9af0187469, State: Initialized, Role: FOLLOWER
I20260929 18:36:41.099437 11101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70893d2ec9754021bf3b2b9af0187469" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:36:41.099568 11101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:41.099637 11101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:41.099738 11101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:41.100744 11101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70893d2ec9754021bf3b2b9af0187469" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:36:41.101078 11101 leader_election.cc:302] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70893d2ec9754021bf3b2b9af0187469; no voters:
I20260929 18:36:41.101373 11101 leader_election.cc:288] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:41.101540 11106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:41.101761 11106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [term 1 LEADER]: Becoming Leader. State: Replica: 70893d2ec9754021bf3b2b9af0187469, State: Running, Role: LEADER
I20260929 18:36:41.102120 11106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70893d2ec9754021bf3b2b9af0187469" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:36:41.102591 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.088986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.0.198.62:42329)
0929 18:36:41.089429 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.089445 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.089482 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:36:41.090180 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.090199 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.090545 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.090831 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:41.091696 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.092809 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.093813 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.094369 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.094518 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.097252 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:41.097257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:41.097271 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:41.097300 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:41.099537 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.100035 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.100040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.100044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.100153 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.100423 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.100425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.100426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.100821 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:41.101032 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.101352 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.101689 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":136,"threads_started":1}
I20260929 18:36:41.102700 11101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:41.103710 11108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70893d2ec9754021bf3b2b9af0187469. Latest consensus state: current_term: 1 leader_uuid: "70893d2ec9754021bf3b2b9af0187469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70893d2ec9754021bf3b2b9af0187469" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } } }
I20260929 18:36:41.103832 11108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:41.104105 11107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70893d2ec9754021bf3b2b9af0187469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70893d2ec9754021bf3b2b9af0187469" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } } }
I20260929 18:36:41.104190 11107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:41.104737 11115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:41.107616 11115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:41.112775 11115 catalog_manager.cc:1347] Generated new cluster ID: 35599f753e144e6a8df8b7907efb66cf
I20260929 18:36:41.112911 11115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:41.129683 11115 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:41.130640 11115 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:41.134542 11115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469: Generated new TSK 0
I20260929 18:36:41.135327 11115 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:36:41.169787 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:41.281661 11127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:41.281948 11127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:41.282017 11127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:41.286315 11127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:41.286374 11127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:41.286438 11127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:36:41.291960 11127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:41.293146 11127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:41.294283 11127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:41.304150 11135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:41.304220 11127 server_base.cc:1034] running on GCE node
W20260929 18:36:41.304171 11132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.304152 11133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:41.304831 11127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:41.305481 11127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:41.306651 11127 hybrid_clock.cc:648] HybridClock initialized: now 1790707001306629 us; error 42 us; skew 500 ppm
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:41.309319 11127 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:36:41.310287 11127 webserver.cc:460] Webserver started at http://127.0.198.1:34699/ using document root <none> and password file <none>
I20260929 18:36:41.310822 11127 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:41.310887 11127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:41.311074 11127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:41.313338 11127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "7fcdc0faac864895aa69527b0a9759b1"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "c237ca2f16649a3bcfdd7ebbb607c075"
server_key_iv: "148474b950221c85f5984f949969dda1"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.313907 11127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "7fcdc0faac864895aa69527b0a9759b1"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "c237ca2f16649a3bcfdd7ebbb607c075"
server_key_iv: "148474b950221c85f5984f949969dda1"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.317628 11127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:41.320279 11142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.321542 11127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:41.321664 11127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "7fcdc0faac864895aa69527b0a9759b1"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "c237ca2f16649a3bcfdd7ebbb607c075"
server_key_iv: "148474b950221c85f5984f949969dda1"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.321776 11127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:41.335572 11127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:41.338754 11127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:41.338945 11127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:41.339530 11127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:41.340348 11127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:41.340394 11127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.340445 11127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:41.340473 11127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.348937 11127 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:36277
I20260929 18:36:41.348963 11255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:36277 every 8 connection(s)
I20260929 18:36:41.349809 11127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:41.356688 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 11127
I20260929 18:36:41.356782 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:41.357033 792 external_mini_cluster.cc:1428] Setting key e81de0053c4eb011e5f754919c2dea5f
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:41.363694 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.351711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:47455 (local address 127.0.198.62:42329)
0929 18:36:41.351873 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.351878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.353190 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 18:36:41.354072 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.354075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.354125 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.354197 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:41.355333 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.355849 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.357070 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.357323 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.357421 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.360110 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:41.360113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:41.360115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:41.360139 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:41.361489 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.362133 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.362135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.362136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.362211 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.362573 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.362575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.362576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.362749 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:41.362816 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.363469 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.363574 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:36:41.364917 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.352156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:42329 (local address 127.0.198.1:47455)
0929 18:36:41.353042 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.353085 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.353857 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.354356 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.354363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.354717 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:41.355214 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.355227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.355979 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.355983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.356955 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.356964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.357468 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.357471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.357653 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.358258 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:41.358273 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:41.359945 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:41.361611 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:41.361618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:41.361637 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:41.362017 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:41.362294 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:41.362297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:41.362300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:41.362489 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:41.362844 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:41.362851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:41.363214 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.363435 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.363807 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":527,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:36:41.366343 11256 heartbeater.cc:347] Connected to a master server at 127.0.198.62:42329
I20260929 18:36:41.366580 11256 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:41.367086 11256 heartbeater.cc:511] Master 127.0.198.62:42329 requested a full tablet report, sending...
I20260929 18:36:41.368664 11065 ts_manager.cc:194] Registered new tserver with Master: 7fcdc0faac864895aa69527b0a9759b1 (127.0.198.1:36277)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:36:41.370036 11065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:47455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:36:41.385769 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:41.494565 11261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:41.494835 11261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:41.494877 11261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:41.498811 11261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:41.498862 11261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:41.498926 11261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:36:41.504356 11261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:41.505352 11261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:41.506369 11261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:41.515993 11261 server_base.cc:1034] running on GCE node
W20260929 18:36:41.515944 11267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.515965 11269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.515942 11266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:41.516587 11261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:41.517223 11261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:41.518404 11261 hybrid_clock.cc:648] HybridClock initialized: now 1790707001518379 us; error 40 us; skew 500 ppm
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:41.521116 11261 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:36:41.522073 11261 webserver.cc:460] Webserver started at http://127.0.198.2:46391/ using document root <none> and password file <none>
I20260929 18:36:41.522603 11261 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:41.522666 11261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:41.522855 11261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:41.524839 11261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "5047c26760004972a8683b3aabe84a37"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "9e7dfe8ccde5f39cfdc51371cd81001c"
server_key_iv: "5e0a75d6cbbe285e762e18f8768a83df"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.525429 11261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "5047c26760004972a8683b3aabe84a37"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "9e7dfe8ccde5f39cfdc51371cd81001c"
server_key_iv: "5e0a75d6cbbe285e762e18f8768a83df"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.529220 11261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:36:41.531927 11276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.533236 11261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:36:41.533349 11261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "5047c26760004972a8683b3aabe84a37"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "9e7dfe8ccde5f39cfdc51371cd81001c"
server_key_iv: "5e0a75d6cbbe285e762e18f8768a83df"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.533463 11261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:41.547669 11261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:41.550858 11261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:41.551062 11261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:41.551648 11261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:41.552484 11261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:41.552532 11261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.552584 11261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:41.552613 11261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.560731 11261 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:40175
I20260929 18:36:41.560758 11389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:40175 every 8 connection(s)
I20260929 18:36:41.561668 11261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:41.562219 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 11261
I20260929 18:36:41.562326 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:41.562654 792 external_mini_cluster.cc:1428] Setting key b457d4a6e7cfd9b6d7ef395be7ab2a36
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:41.578341 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.563400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:35471 (local address 127.0.198.62:42329)
0929 18:36:41.563517 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.563520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.564525 (+ 1005us) server_negotiation.cc:407] Connection header received
0929 18:36:41.565804 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.565807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.565860 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.565938 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:41.567120 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.567708 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.568836 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.569059 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.569145 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.574845 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:41.574849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:41.574852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:41.574882 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:41.576204 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.576754 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.576757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.576759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.576839 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.577211 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.577214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.577216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.577407 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:41.577501 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.578050 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.578181 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:36:41.579202 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.563895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:42329 (local address 127.0.198.2:35471)
0929 18:36:41.564348 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.564399 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.565606 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.566138 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.566146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.566499 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:41.566984 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.566998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.567824 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.567828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.568724 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.568733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.569195 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.569199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.569368 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.570024 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:41.570043 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:41.574641 (+ 4598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:41.576325 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:41.576331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:41.576350 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:41.576646 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:41.576961 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:41.576964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:41.576966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:41.577121 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:41.577509 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:41.577516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:41.577793 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.577997 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.578374 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260929 18:36:41.580446 11390 heartbeater.cc:347] Connected to a master server at 127.0.198.62:42329
I20260929 18:36:41.580670 11390 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:41.581144 11390 heartbeater.cc:511] Master 127.0.198.62:42329 requested a full tablet report, sending...
I20260929 18:36:41.582310 11065 ts_manager.cc:194] Registered new tserver with Master: 5047c26760004972a8683b3aabe84a37 (127.0.198.2:40175)
I20260929 18:36:41.582863 11065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:35471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:36:41.589711 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:41.699110 11395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:41.699386 11395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:41.699430 11395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:41.703548 11395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:41.703622 11395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:41.703688 11395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:36:41.709069 11395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:36:41.710168 11395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:41.711218 11395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:41.720889 11400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.720889 11403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:41.721067 11395 server_base.cc:1034] running on GCE node
W20260929 18:36:41.720893 11401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:41.721622 11395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:41.722304 11395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:41.723469 11395 hybrid_clock.cc:648] HybridClock initialized: now 1790707001723449 us; error 41 us; skew 500 ppm
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:41.726256 11395 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:36:41.727214 11395 webserver.cc:460] Webserver started at http://127.0.198.3:42995/ using document root <none> and password file <none>
I20260929 18:36:41.727756 11395 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:41.727819 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:41.728005 11395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:41.730118 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "505e4655a25a4e928a3abbe5c19de2da"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "a935a9ba3b6e5043ff602c0f6a0c4da5"
server_key_iv: "f229ab52ab712443f2b4925e95435e4d"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.730679 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "505e4655a25a4e928a3abbe5c19de2da"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "a935a9ba3b6e5043ff602c0f6a0c4da5"
server_key_iv: "f229ab52ab712443f2b4925e95435e4d"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.734449 11395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:41.736845 11410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.738248 11395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:41.738353 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "505e4655a25a4e928a3abbe5c19de2da"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-q5qx"
server_key: "a935a9ba3b6e5043ff602c0f6a0c4da5"
server_key_iv: "f229ab52ab712443f2b4925e95435e4d"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.738467 11395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:41.755872 11395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:41.759086 11395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:41.759260 11395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:41.759862 11395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:41.760671 11395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:41.760717 11395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.760756 11395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:41.760780 11395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.769008 11395 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:39877
I20260929 18:36:41.769068 11523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:39877 every 8 connection(s)
I20260929 18:36:41.769908 11395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:41.775990 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 11395
I20260929 18:36:41.776098 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:41.776346 792 external_mini_cluster.cc:1428] Setting key 831f839011447a69d54a06254026678f
Sep 29 18:36:41 dist-test-slave-q5qx krb5kdc[11020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:36:41.784157 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.771809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:52533 (local address 127.0.198.62:42329)
0929 18:36:41.772006 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.772011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.772920 (+ 909us) server_negotiation.cc:407] Connection header received
0929 18:36:41.774283 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.774286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.774334 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.774415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:41.775664 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.776295 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.777869 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.778153 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.778249 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.780826 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:41.780829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:41.780831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:41.780862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:41.782160 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.782665 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.782668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.782669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.782752 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.783151 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.783154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.783156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.783353 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:41.783479 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.783928 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.784034 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:36:41.784958 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.772202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:42329 (local address 127.0.198.3:52533)
0929 18:36:41.772755 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.772807 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.774059 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.774631 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.774639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.775017 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:41.775506 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.775521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.776520 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.776526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.777747 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.777760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.778300 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.778304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.778511 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.779195 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:41.779212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:41.780647 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:41.782259 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:41.782264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:41.782277 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:41.782555 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:41.782871 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:41.782874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:41.782875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:41.783025 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:41.783441 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:41.783447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:41.783696 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.783854 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.784134 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":182,"threads_started":1}
I20260929 18:36:41.786180 11524 heartbeater.cc:347] Connected to a master server at 127.0.198.62:42329
I20260929 18:36:41.786422 11524 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:41.787006 11524 heartbeater.cc:511] Master 127.0.198.62:42329 requested a full tablet report, sending...
I20260929 18:36:41.788034 11065 ts_manager.cc:194] Registered new tserver with Master: 505e4655a25a4e928a3abbe5c19de2da (127.0.198.3:39877)
I20260929 18:36:41.788599 11065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:52533
I20260929 18:36:41.790355 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:41.799086 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.792137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.198.62:42329)
0929 18:36:41.792283 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.792287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.792481 (+ 194us) server_negotiation.cc:407] Connection header received
0929 18:36:41.792642 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.792645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.792685 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.792756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:41.793421 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.793924 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.794780 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.794998 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.795079 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.796350 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:41.796354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:41.796357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:41.796389 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:41.797530 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.797940 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.797943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.797945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.798056 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:41.798286 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:41.798289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:41.798291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:41.798502 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:41.798625 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.798829 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.798937 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:36:41.801837 11065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:41.804859 11065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:41.818958 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.813645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:40175 (local address 127.0.0.1:58896)
0929 18:36:41.813914 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.813926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.814030 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.814738 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.814740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.814775 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:41.815011 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.815019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.816483 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.816486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.817241 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.817250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.818386 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.818390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.818495 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.818536 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.818663 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.818793 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":54,"spinlock_wait_cycles":16256,"thread_start_us":85,"threads_started":1}
I20260929 18:36:41.818967 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.813907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.0.198.2:40175)
0929 18:36:41.814342 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.814352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.814368 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:36:41.814445 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.814449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.814582 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.815138 (+ 556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:41.815301 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.816359 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.817367 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.818279 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.818398 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.818542 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.818708 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.818808 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":174,"threads_started":1}
I20260929 18:36:41.818997 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.812952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39877 (local address 127.0.0.1:59742)
0929 18:36:41.813398 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.813445 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.813569 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.814724 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.814729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.814773 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:41.814998 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.815006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.816126 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.816130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.817303 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.817312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.818621 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.818624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.818718 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.818742 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.818811 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.818877 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":70,"spinlock_wait_cycles":23808,"thread_start_us":80,"threads_started":1}
I20260929 18:36:41.819423 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.813156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.0.198.3:39877)
0929 18:36:41.814245 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.814258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.814282 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:36:41.814358 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.814363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.814576 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.814721 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:41.815132 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.816010 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.817404 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.818279 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.818390 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.818542 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.819171 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.819273 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":123,"threads_started":1}
I20260929 18:36:41.819906 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.813080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36277 (local address 127.0.0.1:45344)
0929 18:36:41.813527 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.813541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.813630 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.814731 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.814734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.814776 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:41.815013 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.815019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.816385 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.816388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.818152 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.818161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.819575 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.819579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.819661 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.819683 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.819728 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.819790 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":50,"spinlock_wait_cycles":13696,"thread_start_us":118,"threads_started":1}
I20260929 18:36:41.820595 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.813715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.0.198.1:36277)
0929 18:36:41.814264 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.814275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.814297 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:36:41.814369 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.814374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.814575 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.814949 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:41.815089 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.816261 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.818278 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.819448 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.820171 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.820315 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.820365 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.820455 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":93,"threads_started":1}
I20260929 18:36:41.822400 11324 tablet_service.cc:1467] Processing CreateTablet for tablet b165e46af9bf4d3e87a2fe6beba3d81d (DEFAULT_TABLE table=test-table [id=678c99c0ccd34a728676634dde1edb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:41.822400 11458 tablet_service.cc:1467] Processing CreateTablet for tablet b165e46af9bf4d3e87a2fe6beba3d81d (DEFAULT_TABLE table=test-table [id=678c99c0ccd34a728676634dde1edb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:41.822731 11190 tablet_service.cc:1467] Processing CreateTablet for tablet b165e46af9bf4d3e87a2fe6beba3d81d (DEFAULT_TABLE table=test-table [id=678c99c0ccd34a728676634dde1edb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:41.823537 11190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b165e46af9bf4d3e87a2fe6beba3d81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:41.823535 11458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b165e46af9bf4d3e87a2fe6beba3d81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:41.823532 11324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b165e46af9bf4d3e87a2fe6beba3d81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:41.829855 11541 tablet_bootstrap.cc:492] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: Bootstrap starting.
I20260929 18:36:41.831499 11543 tablet_bootstrap.cc:492] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: Bootstrap starting.
I20260929 18:36:41.831679 11542 tablet_bootstrap.cc:492] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Bootstrap starting.
I20260929 18:36:41.832161 11541 tablet_bootstrap.cc:654] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:41.833169 11541 log.cc:824] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:41.834163 11543 tablet_bootstrap.cc:654] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:41.834173 11542 tablet_bootstrap.cc:654] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:41.835019 11543 log.cc:824] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:41.835019 11542 log.cc:824] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:41.836198 11541 tablet_bootstrap.cc:492] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: No bootstrap required, opened a new log
I20260929 18:36:41.836400 11541 ts_tablet_manager.cc:1397] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:36:41.836946 11542 tablet_bootstrap.cc:492] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: No bootstrap required, opened a new log
I20260929 18:36:41.837128 11542 ts_tablet_manager.cc:1397] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:36:41.837606 11543 tablet_bootstrap.cc:492] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: No bootstrap required, opened a new log
I20260929 18:36:41.837781 11543 ts_tablet_manager.cc:1397] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:36:41.839443 11541 raft_consensus.cc:348] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.839658 11541 raft_consensus.cc:374] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:41.839709 11541 raft_consensus.cc:729] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fcdc0faac864895aa69527b0a9759b1, State: Initialized, Role: FOLLOWER
I20260929 18:36:41.840122 11541 consensus_queue.cc:260] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.840756 11256 heartbeater.cc:503] Master 127.0.198.62:42329 was elected leader, sending a full tablet report...
I20260929 18:36:41.840862 11542 raft_consensus.cc:348] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.841094 11541 ts_tablet_manager.cc:1428] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:36:41.841130 11542 raft_consensus.cc:374] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:41.841184 11542 raft_consensus.cc:729] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505e4655a25a4e928a3abbe5c19de2da, State: Initialized, Role: FOLLOWER
I20260929 18:36:41.841357 11543 raft_consensus.cc:348] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.841604 11543 raft_consensus.cc:374] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:41.841656 11543 raft_consensus.cc:729] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5047c26760004972a8683b3aabe84a37, State: Initialized, Role: FOLLOWER
I20260929 18:36:41.841620 11542 consensus_queue.cc:260] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.842401 11543 consensus_queue.cc:260] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.842635 11542 ts_tablet_manager.cc:1428] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:36:41.842989 11524 heartbeater.cc:503] Master 127.0.198.62:42329 was elected leader, sending a full tablet report...
I20260929 18:36:41.843245 11543 ts_tablet_manager.cc:1428] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260929 18:36:41.843544 11390 heartbeater.cc:503] Master 127.0.198.62:42329 was elected leader, sending a full tablet report...
W20260929 18:36:41.851017 11257 tablet.cc:2369] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:41.864391 11549 raft_consensus.cc:482] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:41.864588 11549 raft_consensus.cc:504] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.865784 11549 leader_election.cc:288] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 505e4655a25a4e928a3abbe5c19de2da (127.0.198.3:39877), 7fcdc0faac864895aa69527b0a9759b1 (127.0.198.1:36277)
I20260929 18:36:41.870872 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.866194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:57435 (local address 127.0.198.3:39877)
0929 18:36:41.866306 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.866310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.866609 (+ 299us) server_negotiation.cc:407] Connection header received
0929 18:36:41.866695 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.866698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.866744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.866802 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:41.867035 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.867760 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.869528 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.870183 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.870297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.870364 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.870688 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.870759 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:41.870872 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.866175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:55871 (local address 127.0.198.1:36277)
0929 18:36:41.866308 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.866313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.866330 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:36:41.866488 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.866491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.866536 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.866617 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:41.866873 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.867723 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.869528 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.870183 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.870297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.870364 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.870655 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.870744 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:36:41.870878 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.866082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36277 (local address 127.0.198.2:55871)
0929 18:36:41.866268 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.866283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.866376 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.866642 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.866645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.866660 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:41.866772 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.866777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.867861 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.867865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.869331 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.869343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.870326 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.870330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.870472 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.870523 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.870600 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.870713 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":166}
I20260929 18:36:41.870885 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.866198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39877 (local address 127.0.198.2:57435)
0929 18:36:41.866518 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.866528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:41.866612 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:41.866832 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:41.866835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:41.866850 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:41.866941 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.866946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.867861 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.867865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.869331 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:41.869343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.870326 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.870330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:41.870467 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.870513 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:36:41.870600 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:36:41.870712 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":114,"thread_start_us":130,"threads_started":1}
I20260929 18:36:41.871639 11478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165e46af9bf4d3e87a2fe6beba3d81d" candidate_uuid: "5047c26760004972a8683b3aabe84a37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505e4655a25a4e928a3abbe5c19de2da" is_pre_election: true
I20260929 18:36:41.871627 11210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165e46af9bf4d3e87a2fe6beba3d81d" candidate_uuid: "5047c26760004972a8683b3aabe84a37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fcdc0faac864895aa69527b0a9759b1" is_pre_election: true
I20260929 18:36:41.871908 11210 raft_consensus.cc:2454] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5047c26760004972a8683b3aabe84a37 in term 0.
I20260929 18:36:41.871912 11478 raft_consensus.cc:2454] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5047c26760004972a8683b3aabe84a37 in term 0.
I20260929 18:36:41.872344 11279 leader_election.cc:302] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5047c26760004972a8683b3aabe84a37, 7fcdc0faac864895aa69527b0a9759b1; no voters:
I20260929 18:36:41.872610 11549 raft_consensus.cc:2781] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:41.872684 11549 raft_consensus.cc:482] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:41.872715 11549 raft_consensus.cc:3037] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:41.873751 11549 raft_consensus.cc:504] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.874233 11549 leader_election.cc:288] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [CANDIDATE]: Term 1 election: Requested vote from peers 505e4655a25a4e928a3abbe5c19de2da (127.0.198.3:39877), 7fcdc0faac864895aa69527b0a9759b1 (127.0.198.1:36277)
I20260929 18:36:41.874567 11478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165e46af9bf4d3e87a2fe6beba3d81d" candidate_uuid: "5047c26760004972a8683b3aabe84a37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505e4655a25a4e928a3abbe5c19de2da"
I20260929 18:36:41.874696 11478 raft_consensus.cc:3037] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:41.875700 11478 raft_consensus.cc:2454] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5047c26760004972a8683b3aabe84a37 in term 1.
I20260929 18:36:41.875929 11210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165e46af9bf4d3e87a2fe6beba3d81d" candidate_uuid: "5047c26760004972a8683b3aabe84a37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fcdc0faac864895aa69527b0a9759b1"
I20260929 18:36:41.875975 11278 leader_election.cc:302] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5047c26760004972a8683b3aabe84a37, 505e4655a25a4e928a3abbe5c19de2da; no voters:
I20260929 18:36:41.876047 11210 raft_consensus.cc:3037] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:41.876133 11549 raft_consensus.cc:2781] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:41.876349 11549 raft_consensus.cc:686] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 1 LEADER]: Becoming Leader. State: Replica: 5047c26760004972a8683b3aabe84a37, State: Running, Role: LEADER
I20260929 18:36:41.876698 11549 consensus_queue.cc:237] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } }
I20260929 18:36:41.877161 11210 raft_consensus.cc:2454] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5047c26760004972a8683b3aabe84a37 in term 1.
I20260929 18:36:41.879834 11063 catalog_manager.cc:5495] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5047c26760004972a8683b3aabe84a37 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "5047c26760004972a8683b3aabe84a37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5047c26760004972a8683b3aabe84a37" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 40175 } health_report { overall_health: HEALTHY } } }
I20260929 18:36:41.899133 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.895567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.0.198.2:40175)
0929 18:36:41.895696 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.895701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.895715 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:36:41.895864 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.895867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.895948 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.896058 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:41.896293 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.896786 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.897628 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.897850 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.897937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.898269 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:41.898712 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:41.898818 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.898891 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.898963 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:36:41.905305 11478 raft_consensus.cc:1261] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 1 FOLLOWER]: Refusing update from remote peer 5047c26760004972a8683b3aabe84a37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:41.905305 11210 raft_consensus.cc:1261] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 1 FOLLOWER]: Refusing update from remote peer 5047c26760004972a8683b3aabe84a37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:41.906064 11551 consensus_queue.cc:1035] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "505e4655a25a4e928a3abbe5c19de2da" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:41.906268 11549 consensus_queue.cc:1035] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fcdc0faac864895aa69527b0a9759b1" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:41.913195 11554 mvcc.cc:205] Tried to move back new op lower bound from 7334735879796432896 to 7334735879689736192. Current Snapshot: MvccSnapshot[applied={T|T < 7334735879796432896}]
I20260929 18:36:41.922391 11064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:41.922605 11064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:41.925151 11064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70893d2ec9754021bf3b2b9af0187469: Sending DeleteTablet for 3 replicas of tablet b165e46af9bf4d3e87a2fe6beba3d81d
I20260929 18:36:41.926069 11458 tablet_service.cc:1514] Processing DeleteTablet for tablet b165e46af9bf4d3e87a2fe6beba3d81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:41 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:59742
I20260929 18:36:41.926069 11324 tablet_service.cc:1514] Processing DeleteTablet for tablet b165e46af9bf4d3e87a2fe6beba3d81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:41 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:58896
I20260929 18:36:41.926121 11190 tablet_service.cc:1514] Processing DeleteTablet for tablet b165e46af9bf4d3e87a2fe6beba3d81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:41 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:45344
I20260929 18:36:41.926620 11565 tablet_replica.cc:330] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1: stopping tablet replica
I20260929 18:36:41.926805 11564 tablet_replica.cc:330] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: stopping tablet replica
I20260929 18:36:41.926964 11565 raft_consensus.cc:2229] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:41.927183 11564 raft_consensus.cc:2229] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:41.927393 11565 raft_consensus.cc:2258] T b165e46af9bf4d3e87a2fe6beba3d81d P 7fcdc0faac864895aa69527b0a9759b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:41.927457 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 11127
I20260929 18:36:41.928921 11564 raft_consensus.cc:2258] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:41.930428 11564 ts_tablet_manager.cc:1905] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:41.932600 11563 tablet_replica.cc:330] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: stopping tablet replica
I20260929 18:36:41.932796 11564 ts_tablet_manager.cc:1918] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:41.932960 11564 log.cc:1196] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/b165e46af9bf4d3e87a2fe6beba3d81d
I20260929 18:36:41.932991 11563 raft_consensus.cc:2229] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:41.933404 11564 ts_tablet_manager.cc:1939] T b165e46af9bf4d3e87a2fe6beba3d81d P 505e4655a25a4e928a3abbe5c19de2da: Deleting consensus metadata
I20260929 18:36:41.933493 11563 raft_consensus.cc:2258] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:41.934968 11563 ts_tablet_manager.cc:1905] T b165e46af9bf4d3e87a2fe6beba3d81d P 5047c26760004972a8683b3aabe84a37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:41.935248 11050 catalog_manager.cc:4841] TS 505e4655a25a4e928a3abbe5c19de2da (127.0.198.3:39877): tablet b165e46af9bf4d3e87a2fe6beba3d81d (table test-table [id=678c99c0ccd34a728676634dde1edb08]) successfully deleted
W20260929 18:36:41.935273 11051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:36277 (error 108)
I20260929 18:36:41.935662 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 11261
I20260929 18:36:41.937861 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.937591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36277 (local address 127.0.0.1:45352)
0929 18:36:41.937726 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.937786 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:36:41.938052 11051 catalog_manager.cc:4584] TS 7fcdc0faac864895aa69527b0a9759b1 (127.0.198.1:36277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b165e46af9bf4d3e87a2fe6beba3d81d: Network error: Client connection negotiation failed: client connection to 127.0.198.1:36277: connect: Connection refused (error 111)
I20260929 18:36:41.940804 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 11395
I20260929 18:36:41.941722 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.940946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:40175 (local address 127.0.0.1:58906)
0929 18:36:41.941602 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:41.941648 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.2:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585}
I20260929 18:36:41.945724 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 11033
2026-09-29T18:36:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2130 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:42 dist-test-slave-q5qx krb5kdc[11569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:42 dist-test-slave-q5qx krb5kdc[11569](info): set up 2 sockets
Sep 29 18:36:42 dist-test-slave-q5qx krb5kdc[11569](info): commencing operation
krb5kdc: starting...
W20260929 18:36:42.906807 792 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:36:42 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/rangeradmin_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/rangeradmin_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/rangerlookup_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/rangerlookup_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
I20260929 18:36:43.073273 792 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskal1pt5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/postgres -l logfile start
2026-09-29 18:36:45.666 UTC [11599] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:36:45.666 UTC [11599] LOG: listening on IPv4 address "127.0.198.20", port 37543
2026-09-29 18:36:45.670 UTC [11599] LOG: listening on Unix socket "/tmp/.s.PGSQL.37543"
2026-09-29 18:36:45.698 UTC [11602] LOG: database system was shut down at 2026-09-29 18:36:44 UTC
2026-09-29 18:36:45.701 UTC [11599] LOG: database system is ready to accept connections
I20260929 18:36:46.566736 792 mini_postgres.cc:96] Postgres bound to 37543
2026-09-29 18:36:46.572 UTC [11610] FATAL: database "slave" does not exist
127.0.198.20:37543 - accepting connections
I20260929 18:36:46.573421 792 mini_ranger.cc:161] Starting Ranger...
I20260929 18:36:46.588550 792 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:36:47.104848 792 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:36:47.104948 792 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-admin
2026-09-29 18:36:47,375 [I] DB FLAVOR :POSTGRES
2026-09-29 18:36:47,375 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:36:47,375 [I] Checking connection..
2026-09-29 18:36:47,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:36:47,637 [I] Checking connection passed.
2026-09-29 18:36:47,637 [I] --------- Verifying version history table ---------
2026-09-29 18:36:47,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:36:47,886 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:36:47,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:36:48,126 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:36:48,126 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:36:48,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:36:48.374 UTC [11674] WARNING: there is no transaction in progress
2026-09-29 18:36:48,380 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:36:48,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:36:48,635 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:36:48,635 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:36:48,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:36:48,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:36:49,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q5qx.c.gcp-upstream.internal','N') ;"
2026-09-29 18:36:49,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:36:49,584 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:36:49,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:36:49,839 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:36:49,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:36:50,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:36:50,320 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:36:50,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:36:51.304 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.321 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.338 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.355 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.372 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.391 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.451 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.459 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.474 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.485 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.496 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.511 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.521 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.528 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.536 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:51.544 UTC [11782] WARNING: there is no transaction in progress
2026-09-29 18:36:52,266 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:36:52,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:36:52,494 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:36:52,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:36:52,724 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:36:52,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:36:52,980 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:36:52,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:36:53,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:36:53,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q5qx.c.gcp-upstream.internal';"
2026-09-29 18:36:53,695 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:36:53,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:36:53,943 [I] DB_PATCHES have already been applied
I20260929 18:36:53.948576 792 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-admin:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskal1pt5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:36:53.948657 792 mini_ranger.cc:193] Using host: 127.0.198.20
I20260929 18:36:53.951876 792 mini_ranger.cc:227] Ranger admin URL: http://127.0.198.20:60637
Sep 29, 2026 6:36:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:36:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:36:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:36:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:36:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:36:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:36:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:36:54.946293 792 mini_ranger.cc:160] Time spent starting Ranger: real 8.373s user 0.004s sys 0.005s
Sep 29, 2026 6:37:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:37:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:37:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:37:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:37:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:37:16.359048 792 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/rangerkms_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/rangerkms_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.198.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.198.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:37:16 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707036, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:37:16.443738 792 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:37:16.453935 792 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:37:17.392325 792 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:37:17.392419 792 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms
I20260929 18:37:17.392509 792 mini_ranger_kms.cc:227] Using postgres at 127.0.198.20:37543
2026-09-29 18:37:17,568 [I] DB FLAVOR :POSTGRES
2026-09-29 18:37:17,569 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:37:17,569 [I] Checking connection
2026-09-29 18:37:17,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:37:17,810 [I] connection success
2026-09-29 18:37:17,811 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:37:17,811 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:37:17,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:37:18,054 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:37:18,054 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:37:18,054 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:37:18,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:37543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:37:18,338 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:37:18.814776 792 mini_ranger_kms.cc:310] Created kms service
I20260929 18:37:19.010883 792 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:37:19.053215 792 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:37:19 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707036, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:37:19.465685 792 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:37:19.465848 792 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms:/tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:37:19.465875 792 mini_ranger_kms.cc:240] Using host: 127.0.198.20
I20260929 18:37:19.469197 792 mini_ranger_kms.cc:276] Ranger KMS PID: 11962
I20260929 18:37:19.469334 792 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.198.20:42033
Sep 29, 2026 6:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:37:21.497614 792 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.054s user 0.012s sys 0.148s
I20260929 18:37:21.497758 792 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:37:21.497872 792 mini_ranger_kms.cc:401] 127.0.198.20:42033/kms/v1/keys
Sep 29 18:37:21 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707041, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.198.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:37:22 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707041, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.198.20@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
Sep 29 18:37:22 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707041, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.198.20@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
Sep 29 18:37:24 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:37:24.758141 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41095
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:41095
--ranger_config_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:42033/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:24.873162 12013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:24.873474 12013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:24.873518 12013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:24.877679 12013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:24.877760 12013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:24.877779 12013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:24.877794 12013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:24.877808 12013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:24.883772 12013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.198.20:42033/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:41095
--ranger_config_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:37:24.884879 12013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:24.886248 12013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:37:24.895634 12013 server_base.cc:1034] running on GCE node
W20260929 18:37:24.895584 12018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:24.895567 12021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:24.895758 12019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:24.896548 12013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:24.897698 12013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:24.898859 12013 hybrid_clock.cc:648] HybridClock initialized: now 1790707044898840 us; error 39 us; skew 500 ppm
Sep 29 18:37:24 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:24.905838 12013 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:37:24.906850 12013 webserver.cc:460] Webserver started at http://127.0.198.62:38243/ using document root <none> and password file <none>
I20260929 18:37:24.907384 12013 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:24.907440 12013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:24.907660 12013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:37:24 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:37:24.972499 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "9cd7b4cf47704385aa0e7fc7c2841661"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-q5qx"
server_key: "d280e3585bec4d4d156f5b0d1f8956df"
server_key_iv: "cbc78861a37f3763b45760139924ecc2"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:24.973096 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "9cd7b4cf47704385aa0e7fc7c2841661"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-q5qx"
server_key: "d280e3585bec4d4d156f5b0d1f8956df"
server_key_iv: "cbc78861a37f3763b45760139924ecc2"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:24.976936 12013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:37:24.993825 12028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:24.995697 12013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:37:24.995889 12013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "9cd7b4cf47704385aa0e7fc7c2841661"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-q5qx"
server_key: "d280e3585bec4d4d156f5b0d1f8956df"
server_key_iv: "cbc78861a37f3763b45760139924ecc2"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:24.996047 12013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:25.012176 12013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:25.015712 12013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:25.015906 12013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:25.024749 12013 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:41095
I20260929 18:37:25.024773 12080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:41095 every 8 connection(s)
I20260929 18:37:25.025995 12013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:25.028966 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:25.034597 12081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661: Bootstrap starting.
I20260929 18:37:25.035907 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 12013
I20260929 18:37:25.036036 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:37:25.036306 792 external_mini_cluster.cc:1428] Setting key f8aac97271c667673f45712735a37cf5
I20260929 18:37:25.037448 12081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:25.038326 12081 log.cc:824] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:25.040324 12081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661: No bootstrap required, opened a new log
I20260929 18:37:25.044555 12081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41095 } }
I20260929 18:37:25.044827 12081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:25.044929 12081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd7b4cf47704385aa0e7fc7c2841661, State: Initialized, Role: FOLLOWER
I20260929 18:37:25.045512 12081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41095 } }
I20260929 18:37:25.045682 12081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:25.045780 12081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:25.045890 12081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:37:25 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:37:25.047104 12081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41095 } }
I20260929 18:37:25.047468 12081 leader_election.cc:302] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd7b4cf47704385aa0e7fc7c2841661; no voters:
I20260929 18:37:25.047855 12081 leader_election.cc:288] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:25.047955 12086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:25.048380 12086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd7b4cf47704385aa0e7fc7c2841661, State: Running, Role: LEADER
I20260929 18:37:25.048758 12086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41095 } }
I20260929 18:37:25.049293 12081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:25.051015 12086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd7b4cf47704385aa0e7fc7c2841661. Latest consensus state: current_term: 1 leader_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41095 } } }
I20260929 18:37:25.051235 12086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:25.051493 12081 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:37:25.051641 12100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:25.053092 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.037454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.0.198.62:41095)
0929 18:37:25.037953 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.038012 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.038057 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:37:25.038932 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.038955 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.039372 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.039733 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.040605 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.042017 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.043135 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.043929 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.044094 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.047109 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.047116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.047136 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.047171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.049161 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.049685 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.049729 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.049736 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.049887 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.050241 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.050244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.050246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.050789 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.050967 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.051806 (+ 839us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.052090 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":158,"threads_started":1}
I20260929 18:37:25.056658 12087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd7b4cf47704385aa0e7fc7c2841661" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41095 } } }
I20260929 18:37:25.056906 12087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:25.058223 12100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:25.066545 12100 catalog_manager.cc:1347] Generated new cluster ID: f30e52c5d8bd42028ac6efda19e06ac3
I20260929 18:37:25.066613 12100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:25.073899 12100 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:25.075253 12100 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:25.084993 12100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661: Generated new TSK 0
I20260929 18:37:25.085963 12100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:37:25.780619 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:42033/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41095
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:25.944895 12126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:25.945293 12126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:25.945376 12126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:25.951433 12126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:25.951511 12126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:25.951598 12126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:37:25.959499 12126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.198.20:42033/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:37:25.960922 12126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:25.962337 12126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:25.974649 12131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:25.974751 12132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:25.975212 12126 server_base.cc:1034] running on GCE node
W20260929 18:37:25.975382 12134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:25.975883 12126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:25.976706 12126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:25.977881 12126 hybrid_clock.cc:648] HybridClock initialized: now 1790707045977844 us; error 46 us; skew 500 ppm
Sep 29 18:37:25 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:25.981295 12126 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:37:25.982218 12126 webserver.cc:460] Webserver started at http://127.0.198.1:44565/ using document root <none> and password file <none>
I20260929 18:37:25.982748 12126 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:25.982802 12126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:25.982960 12126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:37:25 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:37:25.997555 12126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "b083e884d0494f1aae3c170fcd931cf8"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-q5qx"
server_key: "9a29ca90ce1a7e9a180d6e68a6c67c97"
server_key_iv: "e349fea75742c1fe7b31861aa917aac1"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:25.998420 12126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "b083e884d0494f1aae3c170fcd931cf8"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-q5qx"
server_key: "9a29ca90ce1a7e9a180d6e68a6c67c97"
server_key_iv: "e349fea75742c1fe7b31861aa917aac1"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.003885 12126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:37:26.026420 12143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.028263 12126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:37:26.028476 12126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "b083e884d0494f1aae3c170fcd931cf8"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-q5qx"
server_key: "9a29ca90ce1a7e9a180d6e68a6c67c97"
server_key_iv: "e349fea75742c1fe7b31861aa917aac1"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.028705 12126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:26.072726 12126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:26.076408 12126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:26.076650 12126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:26.077415 12126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:26.078562 12126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:26.078666 12126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.078758 12126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:26.078812 12126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.090204 12126 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:36641
I20260929 18:37:26.091452 12126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:26.099740 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 12126
I20260929 18:37:26.099876 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:26.100207 792 external_mini_cluster.cc:1428] Setting key b003e0bae43054b0322744428cec56bd
I20260929 18:37:26.101945 12258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:36641 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:37:26.126847 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.104154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41095 (local address 127.0.198.1:56607)
0929 18:37:26.107268 (+ 3114us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.107330 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.108368 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.109338 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.109349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.109712 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:26.110481 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.110504 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.111669 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.111673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.113099 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.113110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.113886 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.113890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.114157 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.117611 (+ 3454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:26.117637 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:26.121007 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:26.123308 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:26.123314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:26.123334 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:26.123669 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:26.124019 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:26.124022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:26.124024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:26.124210 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:26.124690 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:26.124697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:26.125098 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.125308 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.125674 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2855,"thread_start_us":153,"threads_started":1}
I20260929 18:37:26.127650 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.105288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:56607 (local address 127.0.198.62:41095)
0929 18:37:26.108832 (+ 3544us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.108837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.108851 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:26.108893 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.108896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.108944 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.109022 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:26.110666 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.111512 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.113240 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.113587 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.113692 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.121277 (+ 7585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:26.121281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:26.121284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:26.121316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:26.123152 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.123789 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.123793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.123795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.123921 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.124307 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.124310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.124312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.124542 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:26.124633 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.127333 (+ 2700us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.127475 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3462,"thread_start_us":93,"threads_started":1}
I20260929 18:37:26.129581 12259 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41095
I20260929 18:37:26.129824 12259 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:26.130384 12259 heartbeater.cc:511] Master 127.0.198.62:41095 requested a full tablet report, sending...
I20260929 18:37:26.131985 12045 ts_manager.cc:194] Registered new tserver with Master: b083e884d0494f1aae3c170fcd931cf8 (127.0.198.1:36641)
I20260929 18:37:26.133178 12045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:56607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:37:26.138862 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:42033/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41095
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:37:26.287395 12265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:26.287825 12265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:26.287892 12265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:26.294454 12265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:26.294549 12265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:26.294668 12265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:37:26.303547 12265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.198.20:42033/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:37:26.305178 12265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:26.306628 12265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:26.319212 12272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:26.319607 12265 server_base.cc:1034] running on GCE node
W20260929 18:37:26.319239 12274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:26.319424 12271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:26.320327 12265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:26.321187 12265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:26.322376 12265 hybrid_clock.cc:648] HybridClock initialized: now 1790707046322333 us; error 53 us; skew 500 ppm
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:26.326048 12265 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:37:26.327404 12265 webserver.cc:460] Webserver started at http://127.0.198.2:44655/ using document root <none> and password file <none>
I20260929 18:37:26.328172 12265 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:26.328293 12265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:26.328538 12265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:37:26.342545 12265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "0ada8989e223497e9d0c40064e44846c"
format_stamp: "Formatted at 2026-09-29 18:37:26 on dist-test-slave-q5qx"
server_key: "e5f241d7c13b319adbe3aaab0391b2af"
server_key_iv: "3ac44ab88b27824b2dac9952e711bcab"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.343194 12265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "0ada8989e223497e9d0c40064e44846c"
format_stamp: "Formatted at 2026-09-29 18:37:26 on dist-test-slave-q5qx"
server_key: "e5f241d7c13b319adbe3aaab0391b2af"
server_key_iv: "3ac44ab88b27824b2dac9952e711bcab"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.351317 12265 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260929 18:37:26.372776 12281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.374545 12265 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 18:37:26.374677 12265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "0ada8989e223497e9d0c40064e44846c"
format_stamp: "Formatted at 2026-09-29 18:37:26 on dist-test-slave-q5qx"
server_key: "e5f241d7c13b319adbe3aaab0391b2af"
server_key_iv: "3ac44ab88b27824b2dac9952e711bcab"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.374807 12265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:26.404661 12265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:26.408178 12265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:26.408365 12265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:26.409096 12265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:26.410080 12265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:26.410199 12265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.410295 12265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:26.410362 12265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.422395 12265 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:39481
I20260929 18:37:26.423584 12265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:26.424398 12394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:39481 every 8 connection(s)
I20260929 18:37:26.427139 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 12265
I20260929 18:37:26.427268 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:26.427593 792 external_mini_cluster.cc:1428] Setting key cfd86bfdeb111bb0f1c9808129bb9885
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:37:26.446245 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.432486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:33861 (local address 127.0.198.62:41095)
0929 18:37:26.432603 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.432608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.432623 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:26.433386 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.433389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.433450 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.433539 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:26.435195 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.436014 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.437693 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.438049 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.438149 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.441815 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:26.441819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:26.441822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:26.441855 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:26.443653 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.444344 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.444348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.444350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.444444 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.444835 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.444839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.444840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.445073 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:26.445169 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.445889 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.446069 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:37:26.447309 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.431367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41095 (local address 127.0.198.2:33861)
0929 18:37:26.431923 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.431980 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.433135 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.433760 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.433770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.434187 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:26.435010 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.435034 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.436145 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.436149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.437556 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.437568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.438329 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.438333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.438548 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.439191 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:26.439211 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:26.441609 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:26.443854 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:26.443861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:26.443882 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:26.444237 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:26.444541 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:26.444544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:26.444546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:26.444742 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:26.445178 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:26.445186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:26.445597 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.445822 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.446206 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":131,"threads_started":1}
I20260929 18:37:26.448805 12395 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41095
I20260929 18:37:26.449083 12395 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:26.449790 12395 heartbeater.cc:511] Master 127.0.198.62:41095 requested a full tablet report, sending...
I20260929 18:37:26.451042 12045 ts_manager.cc:194] Registered new tserver with Master: 0ada8989e223497e9d0c40064e44846c (127.0.198.2:39481)
I20260929 18:37:26.451671 12045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:33861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:37:26.460872 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:42033/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41095
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
W20260929 18:37:26.625293 12400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:26.625769 12400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:26.625872 12400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:26.635610 12400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:26.635792 12400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:26.635941 12400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:37:26.649688 12400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.198.20:42033/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:37:26.652877 12400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:26.654417 12400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:26.668026 12408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:26.668109 12406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:26.668489 12400 server_base.cc:1034] running on GCE node
W20260929 18:37:26.668962 12405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:26.669548 12400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:26.670388 12400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:26.671579 12400 hybrid_clock.cc:648] HybridClock initialized: now 1790707046671550 us; error 44 us; skew 500 ppm
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:26.675379 12400 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:37:26.676533 12400 webserver.cc:460] Webserver started at http://127.0.198.3:41609/ using document root <none> and password file <none>
I20260929 18:37:26.677263 12400 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:26.677356 12400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:26.677583 12400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:37:26.693250 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "244b2c8a4ccc41bd8e024b8c21697653"
format_stamp: "Formatted at 2026-09-29 18:37:26 on dist-test-slave-q5qx"
server_key: "63c49be90aa36972304322c26b267c89"
server_key_iv: "2077c96549ea3ac79d8e71993f5b0a22"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.697461 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "244b2c8a4ccc41bd8e024b8c21697653"
format_stamp: "Formatted at 2026-09-29 18:37:26 on dist-test-slave-q5qx"
server_key: "63c49be90aa36972304322c26b267c89"
server_key_iv: "2077c96549ea3ac79d8e71993f5b0a22"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.711259 12400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 18:37:26.734418 12415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.736261 12400 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260929 18:37:26.736400 12400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "244b2c8a4ccc41bd8e024b8c21697653"
format_stamp: "Formatted at 2026-09-29 18:37:26 on dist-test-slave-q5qx"
server_key: "63c49be90aa36972304322c26b267c89"
server_key_iv: "2077c96549ea3ac79d8e71993f5b0a22"
server_key_version: "kuduclusterkey@0"
I20260929 18:37:26.736533 12400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:26.767491 12400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:26.771101 12400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:26.771291 12400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:26.771946 12400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:26.773000 12400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:26.773064 12400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.773169 12400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:26.773205 12400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:26.784612 12400 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:42275
I20260929 18:37:26.784652 12528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:42275 every 8 connection(s)
I20260929 18:37:26.785809 12400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:26.790329 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 12400
I20260929 18:37:26.790458 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:26.790723 792 external_mini_cluster.cc:1428] Setting key 49eeb1c3208943581a6908e8410c56a3
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:37:26.804235 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.789105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41095 (local address 127.0.198.3:50727)
0929 18:37:26.789659 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.789718 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.790866 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.791696 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.791707 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.792061 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:26.792705 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.792723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.793886 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.793890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.795329 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.795341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.795972 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.795977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.796196 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.796861 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:26.796882 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:26.798600 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:26.800673 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:26.800679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:26.800698 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:26.801127 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:26.801443 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:26.801446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:26.801448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:26.801645 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:26.802134 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:26.802140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:26.802558 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.802776 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.803140 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":150,"threads_started":1}
I20260929 18:37:26.804819 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.788623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:50727 (local address 127.0.198.62:41095)
0929 18:37:26.788755 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.788759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.789829 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 18:37:26.791143 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.791147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.791207 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.791266 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:26.792873 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.793744 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.795460 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.795804 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.795905 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.798794 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:26.798799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:26.798802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:26.798831 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:26.800545 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.801238 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.801241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.801243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.801355 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.801751 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.801754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.801755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.802039 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:26.802130 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.802851 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.804678 (+ 1827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:37:26.806056 12529 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41095
I20260929 18:37:26.806327 12529 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:26.806914 12529 heartbeater.cc:511] Master 127.0.198.62:41095 requested a full tablet report, sending...
I20260929 18:37:26.808045 12045 ts_manager.cc:194] Registered new tserver with Master: 244b2c8a4ccc41bd8e024b8c21697653 (127.0.198.3:42275)
I20260929 18:37:26.808736 12045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:50727
I20260929 18:37:26.816717 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:37:26.828498 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.819078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.0.198.62:41095)
0929 18:37:26.819204 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.819208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.819467 (+ 259us) server_negotiation.cc:407] Connection header received
0929 18:37:26.819598 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.819601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.819652 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.819732 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:26.820630 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.821441 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.822692 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.823025 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.823167 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.824831 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:26.824835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:26.824838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:26.824869 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:26.826577 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.827086 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.827090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.827091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.827182 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:26.827500 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:26.827504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:26.827506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:26.827740 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:26.827846 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.828214 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.828334 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:37:26.832261 12045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:26.836674 12045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:26.862151 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.850285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:39481 (local address 127.0.0.1:41858)
0929 18:37:26.854357 (+ 4072us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.854371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.854457 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.855281 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.855287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.855313 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:26.855448 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.855454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.856855 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.856858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.859125 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.859137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.861574 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.861578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.861705 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.861745 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.861840 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.861957 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4007,"mutex_wait_us":73,"thread_start_us":78,"threads_started":1}
I20260929 18:37:26.864228 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.849102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:42275 (local address 127.0.0.1:50466)
0929 18:37:26.850993 (+ 1891us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.851049 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.851173 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.859603 (+ 8430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.859606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.859625 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:26.859710 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.859716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.861139 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.861143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.862342 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.862353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.863775 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.863779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.863908 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.863954 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.864006 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.864081 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1689,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260929 18:37:26.864317 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.850464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.198.2:39481)
0929 18:37:26.854701 (+ 4237us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.854714 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.854741 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:37:26.854823 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.854828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.855102 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.855284 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:26.855587 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.856727 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.859687 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.860515 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.860613 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.863952 (+ 3339us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.864044 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.864172 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4126,"thread_start_us":97,"threads_started":1}
I20260929 18:37:26.865239 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.850686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.0.198.3:42275)
0929 18:37:26.858963 (+ 8277us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.858975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.859000 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:37:26.859075 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.859081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.859296 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.859449 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:26.859845 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.861007 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.862518 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.863644 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.864792 (+ 1148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.864952 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.865006 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.865105 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8149,"thread_start_us":122,"threads_started":1}
I20260929 18:37:26.866401 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.849924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36641 (local address 127.0.0.1:40586)
0929 18:37:26.851384 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.851396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.851465 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.856234 (+ 4769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.856237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.856254 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:26.856351 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.856358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.857744 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.857748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.859548 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.859555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.863918 (+ 4363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.863921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.865274 (+ 1353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.865300 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.865373 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.866268 (+ 895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"mutex_wait_us":1728,"thread_start_us":132,"threads_started":1}
I20260929 18:37:26.867087 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.850080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.0.198.1:36641)
0929 18:37:26.855584 (+ 5504us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.855591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.855613 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:37:26.855690 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.855696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.855927 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.856086 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:26.856487 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.857613 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.862781 (+ 5168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.863592 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.863691 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.863853 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.866815 (+ 2962us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.866932 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5395,"thread_start_us":110,"threads_started":1}
I20260929 18:37:26.867643 12321 tablet_service.cc:1467] Processing CreateTablet for tablet 2df4d2f90e4d4cf4a1589bf446f42173 (DEFAULT_TABLE table=test-table [id=d719f52b983b475ca09dad050afed9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:26.868203 12445 tablet_service.cc:1467] Processing CreateTablet for tablet 2df4d2f90e4d4cf4a1589bf446f42173 (DEFAULT_TABLE table=test-table [id=d719f52b983b475ca09dad050afed9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:26.868999 12321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df4d2f90e4d4cf4a1589bf446f42173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:26.869168 12445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df4d2f90e4d4cf4a1589bf446f42173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:26.869936 12193 tablet_service.cc:1467] Processing CreateTablet for tablet 2df4d2f90e4d4cf4a1589bf446f42173 (DEFAULT_TABLE table=test-table [id=d719f52b983b475ca09dad050afed9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:26.871008 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df4d2f90e4d4cf4a1589bf446f42173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:26.878602 12547 tablet_bootstrap.cc:492] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: Bootstrap starting.
I20260929 18:37:26.880056 12548 tablet_bootstrap.cc:492] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Bootstrap starting.
I20260929 18:37:26.882934 12548 tablet_bootstrap.cc:654] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:26.883816 12548 log.cc:824] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:26.885711 12547 tablet_bootstrap.cc:654] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:26.886113 12548 tablet_bootstrap.cc:492] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: No bootstrap required, opened a new log
I20260929 18:37:26.886390 12548 ts_tablet_manager.cc:1397] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:37:26.887146 12547 log.cc:824] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:26.888052 12549 tablet_bootstrap.cc:492] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8: Bootstrap starting.
I20260929 18:37:26.890568 12549 tablet_bootstrap.cc:654] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:26.891539 12549 log.cc:824] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:26.891875 12548 raft_consensus.cc:348] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.892282 12548 raft_consensus.cc:374] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:26.892375 12548 raft_consensus.cc:729] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244b2c8a4ccc41bd8e024b8c21697653, State: Initialized, Role: FOLLOWER
I20260929 18:37:26.893124 12548 consensus_queue.cc:260] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.894070 12547 tablet_bootstrap.cc:492] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: No bootstrap required, opened a new log
I20260929 18:37:26.894295 12547 ts_tablet_manager.cc:1397] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260929 18:37:26.894447 12548 ts_tablet_manager.cc:1428] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:37:26.894841 12529 heartbeater.cc:503] Master 127.0.198.62:41095 was elected leader, sending a full tablet report...
I20260929 18:37:26.898486 12547 raft_consensus.cc:348] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.898818 12547 raft_consensus.cc:374] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:26.898957 12547 raft_consensus.cc:729] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ada8989e223497e9d0c40064e44846c, State: Initialized, Role: FOLLOWER
I20260929 18:37:26.899623 12547 consensus_queue.cc:260] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.900950 12547 ts_tablet_manager.cc:1428] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:37:26.901346 12395 heartbeater.cc:503] Master 127.0.198.62:41095 was elected leader, sending a full tablet report...
I20260929 18:37:26.904922 12549 tablet_bootstrap.cc:492] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8: No bootstrap required, opened a new log
I20260929 18:37:26.905189 12549 ts_tablet_manager.cc:1397] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20260929 18:37:26.911125 12549 raft_consensus.cc:348] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.911569 12549 raft_consensus.cc:374] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:26.911681 12549 raft_consensus.cc:729] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b083e884d0494f1aae3c170fcd931cf8, State: Initialized, Role: FOLLOWER
I20260929 18:37:26.912334 12549 consensus_queue.cc:260] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.913635 12549 ts_tablet_manager.cc:1428] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:37:26.914108 12259 heartbeater.cc:503] Master 127.0.198.62:41095 was elected leader, sending a full tablet report...
I20260929 18:37:26.923725 12554 raft_consensus.cc:482] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:26.924053 12554 raft_consensus.cc:504] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.927103 12554 leader_election.cc:288] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244b2c8a4ccc41bd8e024b8c21697653 (127.0.198.3:42275), b083e884d0494f1aae3c170fcd931cf8 (127.0.198.1:36641)
W20260929 18:37:26.927663 12396 tablet.cc:2369] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:26.930708 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.925675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:42275 (local address 127.0.198.2:33509)
0929 18:37:26.926035 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.926050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.926149 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.926555 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.926558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.926579 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:26.926702 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.926710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.927833 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.927837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.929183 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.929194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.930279 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.930283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.930382 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.930419 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.930495 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.930569 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260929 18:37:26.931062 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.925872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:33509 (local address 127.0.198.3:42275)
0929 18:37:26.926044 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.926060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.926098 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:37:26.926261 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.926264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.926321 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.926422 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:26.926841 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.927672 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.929401 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.930050 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.930151 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.930218 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.930868 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.930952 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:37:26.931733 12465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df4d2f90e4d4cf4a1589bf446f42173" candidate_uuid: "0ada8989e223497e9d0c40064e44846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244b2c8a4ccc41bd8e024b8c21697653" is_pre_election: true
I20260929 18:37:26.932101 12465 raft_consensus.cc:2454] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ada8989e223497e9d0c40064e44846c in term 0.
I20260929 18:37:26.932654 12284 leader_election.cc:302] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ada8989e223497e9d0c40064e44846c, 244b2c8a4ccc41bd8e024b8c21697653; no voters:
I20260929 18:37:26.932940 12554 raft_consensus.cc:2781] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:26.933039 12554 raft_consensus.cc:482] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:26.933090 12554 raft_consensus.cc:3037] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.934333 12554 raft_consensus.cc:504] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.934976 12554 leader_election.cc:288] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [CANDIDATE]: Term 1 election: Requested vote from peers 244b2c8a4ccc41bd8e024b8c21697653 (127.0.198.3:42275), b083e884d0494f1aae3c170fcd931cf8 (127.0.198.1:36641)
I20260929 18:37:26.935293 12465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df4d2f90e4d4cf4a1589bf446f42173" candidate_uuid: "0ada8989e223497e9d0c40064e44846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244b2c8a4ccc41bd8e024b8c21697653"
I20260929 18:37:26.935448 12465 raft_consensus.cc:3037] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.936697 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.926961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:46047 (local address 127.0.198.1:36641)
0929 18:37:26.927068 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.927072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.927551 (+ 479us) server_negotiation.cc:407] Connection header received
0929 18:37:26.927733 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.927736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.927786 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.927830 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:26.929662 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.930439 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.935330 (+ 4891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.935950 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.936061 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.936128 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.936478 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.936575 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:37:26.936795 12465 raft_consensus.cc:2454] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ada8989e223497e9d0c40064e44846c in term 1.
I20260929 18:37:26.937048 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.926567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36641 (local address 127.0.198.2:46047)
0929 18:37:26.927461 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.927473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.927618 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.928048 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.928051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.928067 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:26.928188 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.928195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.931769 (+ 3574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.931772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.935191 (+ 3419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.935201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.936245 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.936248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.936345 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.936367 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.936814 (+ 447us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.936894 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":124,"threads_started":1}
I20260929 18:37:26.937772 12213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df4d2f90e4d4cf4a1589bf446f42173" candidate_uuid: "0ada8989e223497e9d0c40064e44846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b083e884d0494f1aae3c170fcd931cf8" is_pre_election: true
I20260929 18:37:26.938280 12213 raft_consensus.cc:2454] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ada8989e223497e9d0c40064e44846c in term 0.
I20260929 18:37:26.938613 12212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df4d2f90e4d4cf4a1589bf446f42173" candidate_uuid: "0ada8989e223497e9d0c40064e44846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b083e884d0494f1aae3c170fcd931cf8"
I20260929 18:37:26.938716 12212 raft_consensus.cc:3037] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.939357 12284 leader_election.cc:302] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ada8989e223497e9d0c40064e44846c, 244b2c8a4ccc41bd8e024b8c21697653; no voters:
I20260929 18:37:26.939564 12554 raft_consensus.cc:2781] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:26.939781 12554 raft_consensus.cc:686] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 1 LEADER]: Becoming Leader. State: Replica: 0ada8989e223497e9d0c40064e44846c, State: Running, Role: LEADER
I20260929 18:37:26.940001 12212 raft_consensus.cc:2454] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ada8989e223497e9d0c40064e44846c in term 1.
I20260929 18:37:26.940128 12554 consensus_queue.cc:237] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } }
I20260929 18:37:26.943588 12045 catalog_manager.cc:5495] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ada8989e223497e9d0c40064e44846c (127.0.198.2). New cstate: current_term: 1 leader_uuid: "0ada8989e223497e9d0c40064e44846c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ada8989e223497e9d0c40064e44846c" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 39481 } health_report { overall_health: HEALTHY } } }
Sep 29 18:37:26 dist-test-slave-q5qx krb5kdc[11569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:37:26.966915 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.958622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.0.198.2:39481)
0929 18:37:26.958744 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.958748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.961879 (+ 3131us) server_negotiation.cc:407] Connection header received
0929 18:37:26.962176 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.962180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.962263 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.962406 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:26.962675 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.963438 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.964684 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.965002 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.965113 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.965543 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.966158 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.966265 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.966709 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.966775 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:37:26.983743 12212 raft_consensus.cc:1261] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8 [term 1 FOLLOWER]: Refusing update from remote peer 0ada8989e223497e9d0c40064e44846c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.984203 12465 raft_consensus.cc:1261] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 1 FOLLOWER]: Refusing update from remote peer 0ada8989e223497e9d0c40064e44846c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.984627 12558 consensus_queue.cc:1035] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b083e884d0494f1aae3c170fcd931cf8" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.984889 12554 consensus_queue.cc:1035] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [LEADER]: Connected to new peer: Peer: permanent_uuid: "244b2c8a4ccc41bd8e024b8c21697653" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.996276 12561 mvcc.cc:205] Tried to move back new op lower bound from 7334736064433090560 to 7334736064270041088. Current Snapshot: MvccSnapshot[applied={T|T < 7334736064433090560}]
I20260929 18:37:26.999639 12563 mvcc.cc:205] Tried to move back new op lower bound from 7334736064433090560 to 7334736064270041088. Current Snapshot: MvccSnapshot[applied={T|T < 7334736064433090560}]
I20260929 18:37:27.002418 12562 mvcc.cc:205] Tried to move back new op lower bound from 7334736064433090560 to 7334736064270041088. Current Snapshot: MvccSnapshot[applied={T|T < 7334736064433090560}]
I20260929 18:37:27.014371 12043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:27.014577 12043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:27.017944 12043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cd7b4cf47704385aa0e7fc7c2841661: Sending DeleteTablet for 3 replicas of tablet 2df4d2f90e4d4cf4a1589bf446f42173
I20260929 18:37:27.019354 12193 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df4d2f90e4d4cf4a1589bf446f42173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:27 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:40586
I20260929 18:37:27.019912 12321 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df4d2f90e4d4cf4a1589bf446f42173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:27 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:41858
I20260929 18:37:27.020745 12445 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df4d2f90e4d4cf4a1589bf446f42173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:27 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:50466
I20260929 18:37:27.022251 12570 tablet_replica.cc:330] T 2df4d2f90e4d4cf4a1589bf446f42173 P b083e884d0494f1aae3c170fcd931cf8: stopping tablet replica
I20260929 18:37:27.022313 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 12126
I20260929 18:37:27.022985 12571 tablet_replica.cc:330] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c: stopping tablet replica
I20260929 18:37:27.023350 12571 raft_consensus.cc:2229] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:27.023809 12571 raft_consensus.cc:2258] T 2df4d2f90e4d4cf4a1589bf446f42173 P 0ada8989e223497e9d0c40064e44846c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:27.024237 12572 tablet_replica.cc:330] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: stopping tablet replica
I20260929 18:37:27.025465 12572 raft_consensus.cc:2229] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:27.026593 12572 raft_consensus.cc:2258] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:37:27.028697 12032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:36641 (error 108)
I20260929 18:37:27.030443 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 12265
I20260929 18:37:27.032238 12572 ts_tablet_manager.cc:1905] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:27.035039 12572 ts_tablet_manager.cc:1918] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:27.035140 12572 log.cc:1196] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/2df4d2f90e4d4cf4a1589bf446f42173
I20260929 18:37:27.035434 12572 ts_tablet_manager.cc:1939] T 2df4d2f90e4d4cf4a1589bf446f42173 P 244b2c8a4ccc41bd8e024b8c21697653: Deleting consensus metadata
I20260929 18:37:27.036772 12031 catalog_manager.cc:4841] TS 244b2c8a4ccc41bd8e024b8c21697653 (127.0.198.3:42275): tablet 2df4d2f90e4d4cf4a1589bf446f42173 (table test-table [id=d719f52b983b475ca09dad050afed9a2]) successfully deleted
I20260929 18:37:27.037842 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:27.037540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:36641 (local address 127.0.0.1:40602)
0929 18:37:27.037674 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:27.037757 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:36641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:37:27.038044 12032 catalog_manager.cc:4584] TS b083e884d0494f1aae3c170fcd931cf8 (127.0.198.1:36641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2df4d2f90e4d4cf4a1589bf446f42173: Network error: Client connection negotiation failed: client connection to 127.0.198.1:36641: connect: Connection refused (error 111)
W20260929 18:37:27.041489 12542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:27.038345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:39481 (local address 127.0.0.1:41874)
0929 18:37:27.038487 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:27.038501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:27.038590 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:27.041385 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.2:39481: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 18:37:27.044035 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 12400
I20260929 18:37:27.050885 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 12013
I20260929 18:37:27.061625 792 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:37:27.407253 792 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:37:27.407367 792 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:37:27.742552 792 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:37:27.742 UTC [11599] LOG: received smart shutdown request
2026-09-29 18:37:27.746 UTC [11599] LOG: background worker "logical replication launcher" (PID 11608) exited with exit code 1
2026-09-29 18:37:27.747 UTC [11603] LOG: shutting down
2026-09-29 18:37:27.938 UTC [11599] LOG: database system is shut down
2026-09-29T18:37:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46088 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:28 dist-test-slave-q5qx krb5kdc[12583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:28 dist-test-slave-q5qx krb5kdc[12583](info): set up 2 sockets
Sep 29 18:37:28 dist-test-slave-q5qx krb5kdc[12583](info): commencing operation
krb5kdc: starting...
W20260929 18:37:28.997826 792 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:37:29 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/rangeradmin_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/rangeradmin_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/rangerlookup_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/rangerlookup_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
I20260929 18:37:29.172624 792 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskal1pt5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/postgres -l logfile start
2026-09-29 18:37:31.802 UTC [12613] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:37:31.802 UTC [12613] LOG: listening on IPv4 address "127.0.198.20", port 39611
2026-09-29 18:37:31.806 UTC [12613] LOG: listening on Unix socket "/tmp/.s.PGSQL.39611"
2026-09-29 18:37:31.835 UTC [12616] LOG: database system was shut down at 2026-09-29 18:37:30 UTC
2026-09-29 18:37:31.839 UTC [12613] LOG: database system is ready to accept connections
I20260929 18:37:32.705605 792 mini_postgres.cc:96] Postgres bound to 39611
2026-09-29 18:37:32.711 UTC [12624] FATAL: database "slave" does not exist
127.0.198.20:39611 - accepting connections
I20260929 18:37:32.712177 792 mini_ranger.cc:161] Starting Ranger...
I20260929 18:37:32.726045 792 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:37:33.250201 792 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:37:33.250269 792 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-admin
2026-09-29 18:37:33,520 [I] DB FLAVOR :POSTGRES
2026-09-29 18:37:33,521 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:37:33,521 [I] Checking connection..
2026-09-29 18:37:33,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:37:33,763 [I] Checking connection passed.
2026-09-29 18:37:33,763 [I] --------- Verifying version history table ---------
2026-09-29 18:37:33,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:37:34,019 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:37:34,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:37:34,275 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:37:34,275 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:37:34,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:37:34.526 UTC [12688] WARNING: there is no transaction in progress
2026-09-29 18:37:34,531 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:37:34,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:37:34,762 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:37:34,762 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:37:34,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:37:35,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:37:35,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q5qx.c.gcp-upstream.internal','N') ;"
2026-09-29 18:37:35,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:37:35,729 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:37:35,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:37:35,955 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:37:35,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:37:36,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:37:36,442 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:37:36,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:37:37.431 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.448 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.464 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.480 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.496 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.515 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.578 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.586 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.602 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.613 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.624 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.639 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.649 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.657 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.664 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:37.670 UTC [12796] WARNING: there is no transaction in progress
2026-09-29 18:37:38,360 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:37:38,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:37:38,589 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:37:38,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:37:38,829 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:37:38,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:37:39,079 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:37:39,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:37:39,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:37:39,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q5qx.c.gcp-upstream.internal';"
2026-09-29 18:37:39,771 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:37:39,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:37:40,032 [I] DB_PATCHES have already been applied
I20260929 18:37:40.037933 792 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-admin:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskal1pt5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:37:40.038022 792 mini_ranger.cc:193] Using host: 127.0.198.20
I20260929 18:37:40.041218 792 mini_ranger.cc:227] Ranger admin URL: http://127.0.198.20:44225
Sep 29, 2026 6:37:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:37:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:37:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:37:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:37:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:37:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:37:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:37:41.040019 792 mini_ranger.cc:160] Time spent starting Ranger: real 8.328s user 0.003s sys 0.005s
Sep 29, 2026 6:38:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:38:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:38:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:38:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:38:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:38:02.528057 792 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/rangerkms_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/rangerkms_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.198.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.198.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.198.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/HTTP_127.0.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:38:02 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707082, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:38:02.610716 792 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:38:02.620252 792 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:38:03.609668 792 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:38:03.609740 792 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms
I20260929 18:38:03.609759 792 mini_ranger_kms.cc:227] Using postgres at 127.0.198.20:39611
2026-09-29 18:38:03,768 [I] DB FLAVOR :POSTGRES
2026-09-29 18:38:03,768 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:38:03,768 [I] Checking connection
2026-09-29 18:38:03,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:38:03,997 [I] connection success
2026-09-29 18:38:03,997 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:38:03,997 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:38:03,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:38:04,260 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:38:04,260 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:38:04,260 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:38:04,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.198.20:39611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:38:04,524 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:38:04.969594 792 mini_ranger_kms.cc:310] Created kms service
I20260929 18:38:05.156497 792 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:38:05.201061 792 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:38:05 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707082, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:05.612644 792 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:38:05.612804 792 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms:/tmp/dist-test-taskal1pt5/build/debug/bin/postgresql.jar:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskal1pt5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:38:05.612826 792 mini_ranger_kms.cc:240] Using host: 127.0.198.20
I20260929 18:38:05.616822 792 mini_ranger_kms.cc:276] Ranger KMS PID: 12976
I20260929 18:38:05.616902 792 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.198.20:40607
Sep 29, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:38:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:38:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:38:07.684515 792 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.074s user 0.012s sys 0.130s
I20260929 18:38:07.684635 792 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:38:07.684696 792 mini_ranger_kms.cc:401] 127.0.198.20:40607/kms/v1/keys
Sep 29 18:38:07 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707087, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.198.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:38:08 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707087, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.198.20@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
Sep 29 18:38:08 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707087, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.198.20@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
Sep 29 18:38:10 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:38:10.885387 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35269
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:35269
--ranger_config_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:10.992544 13027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:10.992815 13027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:10.992852 13027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:10.996862 13027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:10.996935 13027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:10.996953 13027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:10.996968 13027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:10.996981 13027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:11.002518 13027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.198.20:40607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:35269
--ranger_config_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:11.003594 13027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:11.004594 13027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:11.013386 13032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:11.013423 13033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:11.013590 13027 server_base.cc:1034] running on GCE node
W20260929 18:38:11.013382 13035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:11.014427 13027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:11.016048 13027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:11.017294 13027 hybrid_clock.cc:648] HybridClock initialized: now 1790707091017261 us; error 61 us; skew 500 ppm
Sep 29 18:38:11 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:11.020255 13027 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:11.021190 13027 webserver.cc:460] Webserver started at http://127.0.198.62:40211/ using document root <none> and password file <none>
I20260929 18:38:11.021759 13027 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:11.021819 13027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:11.022065 13027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:38:11 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:11.070160 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "129ddeabf9cc4455ad698f5158e71d90"
format_stamp: "Formatted at 2026-09-29 18:38:11 on dist-test-slave-q5qx"
server_key: "67ad6402be7f6f413ff5209a318a1b91"
server_key_iv: "4c00183397e041bbc6123376548e2f91"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:11.070770 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "129ddeabf9cc4455ad698f5158e71d90"
format_stamp: "Formatted at 2026-09-29 18:38:11 on dist-test-slave-q5qx"
server_key: "67ad6402be7f6f413ff5209a318a1b91"
server_key_iv: "4c00183397e041bbc6123376548e2f91"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:11.075341 13027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:38:11.089488 13042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:11.090867 13027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:38:11.091001 13027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "129ddeabf9cc4455ad698f5158e71d90"
format_stamp: "Formatted at 2026-09-29 18:38:11 on dist-test-slave-q5qx"
server_key: "67ad6402be7f6f413ff5209a318a1b91"
server_key_iv: "4c00183397e041bbc6123376548e2f91"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:11.091132 13027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:11.107097 13027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:11.109951 13027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:11.110175 13027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:11.117728 13027 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:35269
I20260929 18:38:11.117739 13094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:35269 every 8 connection(s)
I20260929 18:38:11.118821 13027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:11.121613 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:11.122514 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13027
I20260929 18:38:11.122697 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:38:11.123026 792 external_mini_cluster.cc:1428] Setting key 4d874e289455456b15df0ab01ba031bb
I20260929 18:38:11.128338 13095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Bootstrap starting.
I20260929 18:38:11.130854 13095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:11.131703 13095 log.cc:824] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:38:11 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:11.133729 13095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: No bootstrap required, opened a new log
I20260929 18:38:11.137077 13095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:11.137367 13095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:11.137425 13095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129ddeabf9cc4455ad698f5158e71d90, State: Initialized, Role: FOLLOWER
I20260929 18:38:11.137898 13095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:11.138088 13095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:11.138154 13095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:11.138280 13095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:11.139385 13095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:11.139714 13095 leader_election.cc:302] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129ddeabf9cc4455ad698f5158e71d90; no voters:
I20260929 18:38:11.140008 13095 leader_election.cc:288] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:11.140158 13100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:11.140192 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:11.124588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.0.198.62:35269)
0929 18:38:11.125058 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:11.125077 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:11.125119 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:38:11.125935 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:11.125958 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:11.126418 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:11.126795 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:11.127676 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.129086 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.130414 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.131097 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.131247 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:11.134229 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:11.134236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:11.134254 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:11.134284 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:11.136866 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:11.137447 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:11.137454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:11.137460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:11.137609 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:11.137902 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:11.137906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:11.137908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:11.138467 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:11.138685 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:11.138935 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 18:38:11.139217 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260929 18:38:11.140836 13100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 LEADER]: Becoming Leader. State: Replica: 129ddeabf9cc4455ad698f5158e71d90, State: Running, Role: LEADER
I20260929 18:38:11.141188 13100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:11.141347 13095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:11.142539 13100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "129ddeabf9cc4455ad698f5158e71d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } } }
I20260929 18:38:11.142967 13100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:11.143198 13100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129ddeabf9cc4455ad698f5158e71d90. Latest consensus state: current_term: 1 leader_uuid: "129ddeabf9cc4455ad698f5158e71d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } } }
I20260929 18:38:11.143330 13100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:11.143390 13114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:11.143486 13095 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:38:11.149152 13114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:11.160609 13114 catalog_manager.cc:1347] Generated new cluster ID: d357c53b99fc4c9482f4e7037fdc554c
I20260929 18:38:11.160688 13114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:11.179363 13114 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:11.180675 13114 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:11.192936 13114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Generated new TSK 0
I20260929 18:38:11.193807 13114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:38:11.845258 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35269
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:11.961848 13140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:11.962137 13140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:11.962182 13140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:11.966339 13140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:11.966409 13140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:11.966490 13140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:38:11.972262 13140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.198.20:40607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:11.973376 13140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:11.974515 13140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:11.984376 13146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:11.984609 13140 server_base.cc:1034] running on GCE node
W20260929 18:38:11.984390 13145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:11.984418 13148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:11.985152 13140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:11.985743 13140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:11.986896 13140 hybrid_clock.cc:648] HybridClock initialized: now 1790707091986874 us; error 73 us; skew 500 ppm
Sep 29 18:38:11 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:11.989718 13140 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:38:11.990810 13140 webserver.cc:460] Webserver started at http://127.0.198.1:37173/ using document root <none> and password file <none>
I20260929 18:38:11.991637 13140 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:11.991731 13140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:11.991992 13140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:38:11 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:12.008431 13140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "ff74c04badfa485c86b0832c7676de23"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "3fd5e132edc640a241da6ab186d90257"
server_key_iv: "86ef29a23fbd70be7cae54fe28f2ac5c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.009302 13140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "ff74c04badfa485c86b0832c7676de23"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "3fd5e132edc640a241da6ab186d90257"
server_key_iv: "86ef29a23fbd70be7cae54fe28f2ac5c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.014671 13140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:38:12.033496 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.034965 13140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:12.035096 13140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "ff74c04badfa485c86b0832c7676de23"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "3fd5e132edc640a241da6ab186d90257"
server_key_iv: "86ef29a23fbd70be7cae54fe28f2ac5c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.035229 13140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:12.073390 13140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:12.076701 13140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:12.076929 13140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:12.077688 13140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:12.078760 13140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:12.078842 13140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.078917 13140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:12.078969 13140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.091925 13140 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:39853
I20260929 18:38:12.093356 13140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:12.094901 13270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:39853 every 8 connection(s)
I20260929 18:38:12.103569 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13140
I20260929 18:38:12.103711 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:38:12.104008 792 external_mini_cluster.cc:1428] Setting key 15ffcb18c7ec6a886bf0409bacf3287d
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:38:12.147361 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.097882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35269 (local address 127.0.198.1:51207)
0929 18:38:12.100633 (+ 2751us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:12.100677 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:12.101440 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:12.102239 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:12.102247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:12.102521 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:12.103137 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.103151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.115253 (+ 12102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.115258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.116850 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.116867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.131039 (+ 14172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.131046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.131337 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.132120 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:12.132144 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:12.133848 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:12.141496 (+ 7648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:12.141503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:12.141523 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:12.141952 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:12.144334 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:12.144338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:12.144341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:12.144654 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:12.145190 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:12.145199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:12.145608 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 18:38:12.145820 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 18:38:12.146245 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2533,"thread_start_us":158,"threads_started":1}
I20260929 18:38:12.148809 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.099126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:51207 (local address 127.0.198.62:35269)
0929 18:38:12.101796 (+ 2670us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:12.101803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:12.101815 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:12.101851 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:12.101853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:12.101892 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:12.101991 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:12.114087 (+ 12096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.115001 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.117042 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.130753 (+ 13711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.132052 (+ 1299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.134112 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:12.134116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:12.134119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:12.134148 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:12.141333 (+ 7185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:12.144069 (+ 2736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:12.144073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:12.144075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:12.144179 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:12.144792 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:12.144796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:12.144798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:12.145056 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:12.148389 (+ 3333us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:12.148485 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:38:12.148644 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2596,"thread_start_us":90,"threads_started":1}
I20260929 18:38:12.150146 13271 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35269
I20260929 18:38:12.150395 13271 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:12.150959 13271 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:38:12.152767 13059 ts_manager.cc:194] Registered new tserver with Master: ff74c04badfa485c86b0832c7676de23 (127.0.198.1:39853)
I20260929 18:38:12.154500 13059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:51207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:38:12.173739 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35269
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:38:12.341863 13279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:12.342291 13279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:12.342409 13279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:12.348711 13279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:12.348795 13279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:12.348881 13279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:38:12.355376 13279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.198.20:40607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:12.356577 13279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:12.357586 13279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:12.368777 13288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:12.368800 13286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:12.369091 13279 server_base.cc:1034] running on GCE node
W20260929 18:38:12.369328 13285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:12.369796 13279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:12.370611 13279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:12.371800 13279 hybrid_clock.cc:648] HybridClock initialized: now 1790707092371772 us; error 42 us; skew 500 ppm
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:12.375290 13279 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:38:12.376377 13279 webserver.cc:460] Webserver started at http://127.0.198.2:43871/ using document root <none> and password file <none>
I20260929 18:38:12.377048 13279 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:12.377130 13279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:12.377336 13279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:12.397228 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "0ab2b09c5e78465ea75a4b1cb3df5474"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "dea55dd3c42d3d01b6fdac4b07cb020a"
server_key_iv: "c22b1301b640e7f2e395ae94fc25046c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.397861 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "0ab2b09c5e78465ea75a4b1cb3df5474"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "dea55dd3c42d3d01b6fdac4b07cb020a"
server_key_iv: "c22b1301b640e7f2e395ae94fc25046c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.402148 13279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:38:12.429033 13295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.430830 13279 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260929 18:38:12.430979 13279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "0ab2b09c5e78465ea75a4b1cb3df5474"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "dea55dd3c42d3d01b6fdac4b07cb020a"
server_key_iv: "c22b1301b640e7f2e395ae94fc25046c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.431128 13279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:12.448908 13279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:12.452661 13279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:12.452864 13279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:12.453601 13279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:12.454695 13279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:12.454764 13279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.454828 13279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:12.454862 13279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.468132 13279 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:45469
I20260929 18:38:12.469379 13279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:12.473626 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13279
I20260929 18:38:12.473766 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:38:12.473873 13408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:45469 every 8 connection(s)
I20260929 18:38:12.474100 792 external_mini_cluster.cc:1428] Setting key f48f77f9ee07172b9cd786612de12820
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:12.492645 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.477734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:39225 (local address 127.0.198.62:35269)
0929 18:38:12.477855 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:12.477859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:12.479294 (+ 1435us) server_negotiation.cc:407] Connection header received
0929 18:38:12.480536 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:12.480541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:12.480594 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:12.480649 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:12.482341 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.483219 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.485013 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.485357 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.485439 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.488526 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:12.488530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:12.488533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:12.488564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:12.490192 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:12.490807 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:12.490811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:12.490813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:12.490901 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:12.491318 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:12.491322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:12.491323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:12.491541 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:12.491622 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:12.492337 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 18:38:12.492476 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:38:12.494434 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.476779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35269 (local address 127.0.198.2:39225)
0929 18:38:12.479125 (+ 2346us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:12.479187 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:12.480286 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:12.481076 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:12.481086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:12.481446 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:12.482168 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.482188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.483411 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.483415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.484863 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.484874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.485647 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.485651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.485860 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.486570 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:12.486590 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:12.488323 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:12.490332 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:12.490338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:12.490356 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:12.490694 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:12.491018 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:12.491021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:12.491023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:12.491211 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:12.491690 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:12.491697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:12.492055 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 18:38:12.492892 (+ 837us) client_negotiation.cc:240] Negotiation successful
0929 18:38:12.493266 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2088,"thread_start_us":175,"threads_started":1}
I20260929 18:38:12.495906 13409 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35269
I20260929 18:38:12.496214 13409 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:12.496809 13409 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
I20260929 18:38:12.497941 13059 ts_manager.cc:194] Registered new tserver with Master: 0ab2b09c5e78465ea75a4b1cb3df5474 (127.0.198.2:45469)
I20260929 18:38:12.498693 13059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:39225
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:38:12.537499 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35269
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:12.699563 13414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:12.699962 13414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:12.700032 13414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:12.706552 13414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:12.706637 13414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:12.706723 13414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:38:12.714496 13414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.198.20:40607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:12.715926 13414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:12.717523 13414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:12.730509 13420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:12.730515 13419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:12.730875 13414 server_base.cc:1034] running on GCE node
W20260929 18:38:12.731233 13422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:12.731673 13414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:12.732409 13414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:12.733590 13414 hybrid_clock.cc:648] HybridClock initialized: now 1790707092733525 us; error 79 us; skew 500 ppm
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:12.736912 13414 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:38:12.737916 13414 webserver.cc:460] Webserver started at http://127.0.198.3:38543/ using document root <none> and password file <none>
I20260929 18:38:12.738615 13414 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:12.738692 13414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:12.738915 13414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:12.763017 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "625b4bacab2002384ad060cdd68aa99f"
server_key_iv: "ec1bdbbb84ba3c79dbc988fce713ef1c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.763935 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "625b4bacab2002384ad060cdd68aa99f"
server_key_iv: "ec1bdbbb84ba3c79dbc988fce713ef1c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.770849 13414 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260929 18:38:12.792088 13429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.793834 13414 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:38:12.794014 13414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "625b4bacab2002384ad060cdd68aa99f"
server_key_iv: "ec1bdbbb84ba3c79dbc988fce713ef1c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:12.794164 13414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:12.815240 13414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:12.818835 13414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:12.819036 13414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:12.819777 13414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:12.822016 13414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:12.822086 13414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.822211 13414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:12.822248 13414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:12.832588 13414 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:45397
I20260929 18:38:12.833750 13414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:12.840983 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13414
I20260929 18:38:12.841115 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:38:12.841456 792 external_mini_cluster.cc:1428] Setting key 48716186810a281260fa4ae7fca083b5
I20260929 18:38:12.861013 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:45397 every 8 connection(s)
Sep 29 18:38:12 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:12.888461 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.861646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:37311 (local address 127.0.198.62:35269)
0929 18:38:12.863243 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:12.863248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:12.863720 (+ 472us) server_negotiation.cc:407] Connection header received
0929 18:38:12.865538 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:12.865542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:12.865621 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:12.865678 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:12.867503 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.868367 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.870203 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.870556 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.870637 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.879948 (+ 9311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:12.879952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:12.879955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:12.879989 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:12.882629 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:12.885257 (+ 2628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:12.885262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:12.885264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:12.885363 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:12.885824 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:12.885828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:12.885829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:12.886082 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:12.886168 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:12.888138 (+ 1970us) server_negotiation.cc:299] Negotiation successful
0929 18:38:12.888280 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1495}
I20260929 18:38:12.890210 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.862137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35269 (local address 127.0.198.3:37311)
0929 18:38:12.863541 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:12.863622 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:12.865299 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:12.866166 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:12.866177 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:12.866547 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:12.867312 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.867336 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.868621 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.868626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.870068 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.870081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.870861 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.870865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.871093 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.875017 (+ 3924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:12.875040 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:12.878835 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:12.883166 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:12.883174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:12.883195 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:12.883631 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:12.885489 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:12.885492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:12.885494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:12.885715 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:12.886237 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:12.886244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:12.886630 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 18:38:12.888654 (+ 2024us) client_negotiation.cc:240] Negotiation successful
0929 18:38:12.889058 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":169,"threads_started":1}
I20260929 18:38:12.891453 13543 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35269
I20260929 18:38:12.891721 13543 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:12.892350 13543 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
I20260929 18:38:12.893508 13059 ts_manager.cc:194] Registered new tserver with Master: 5d1a26466c7644f1b5ac7e6e21e9ddeb (127.0.198.3:45397)
I20260929 18:38:12.894277 13059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:37311
I20260929 18:38:12.903553 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:38:12.904376 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13140
I20260929 18:38:12.911345 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13279
I20260929 18:38:12.918529 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13414
I20260929 18:38:12.925447 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13027
I20260929 18:38:12.933413 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35269
--webserver_interface=127.0.198.62
--webserver_port=40211
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:35269
--ranger_config_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:13.089075 13546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:13.089377 13546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:13.089432 13546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:13.093778 13546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:13.093850 13546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:13.093868 13546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:13.093883 13546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:13.093897 13546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:13.099594 13546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:35269
--ranger_config_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=40211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:13.100718 13546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:13.101725 13546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:13.109855 13551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:13.109854 13552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:13.110062 13554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:13.110256 13546 server_base.cc:1034] running on GCE node
I20260929 18:38:13.110904 13546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:13.111974 13546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:13.113128 13546 hybrid_clock.cc:648] HybridClock initialized: now 1790707093113106 us; error 46 us; skew 500 ppm
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:13.115893 13546 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:13.116792 13546 webserver.cc:460] Webserver started at http://127.0.198.62:40211/ using document root <none> and password file <none>
I20260929 18:38:13.117300 13546 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:13.117369 13546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:13.184048 13546 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.002s sys 0.000s
I20260929 18:38:13.186609 13562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.188141 13546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:13.188284 13546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "129ddeabf9cc4455ad698f5158e71d90"
format_stamp: "Formatted at 2026-09-29 18:38:11 on dist-test-slave-q5qx"
server_key: "67ad6402be7f6f413ff5209a318a1b91"
server_key_iv: "4c00183397e041bbc6123376548e2f91"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:13.188903 13546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:13.207276 13546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:13.210201 13546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:13.210377 13546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:13.217232 13546 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:35269
I20260929 18:38:13.217264 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:35269 every 8 connection(s)
I20260929 18:38:13.218330 13546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:13.222224 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13546
I20260929 18:38:13.222699 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:39853
--local_ip_for_outbound_sockets=127.0.198.1
--tserver_master_addrs=127.0.198.62:35269
--webserver_port=37173
--webserver_interface=127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:13.222735 13615 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:38:13.224706 13615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Bootstrap starting.
I20260929 18:38:13.240864 13615 log.cc:824] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:13.247648 13615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:38:13.248025 13615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Bootstrap complete.
I20260929 18:38:13.251946 13615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:13.252345 13615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129ddeabf9cc4455ad698f5158e71d90, State: Initialized, Role: FOLLOWER
I20260929 18:38:13.252813 13615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:13.252928 13615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:13.252992 13615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:13.253091 13615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:38:13.254981 13615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:13.255303 13615 leader_election.cc:302] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129ddeabf9cc4455ad698f5158e71d90; no voters:
I20260929 18:38:13.255604 13615 leader_election.cc:288] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:38:13.255702 13619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:38:13.255891 13619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [term 2 LEADER]: Becoming Leader. State: Replica: 129ddeabf9cc4455ad698f5158e71d90, State: Running, Role: LEADER
I20260929 18:38:13.256178 13619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } }
I20260929 18:38:13.256798 13615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:13.257092 13619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129ddeabf9cc4455ad698f5158e71d90. Latest consensus state: current_term: 2 leader_uuid: "129ddeabf9cc4455ad698f5158e71d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } } }
I20260929 18:38:13.257201 13619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:13.258198 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "129ddeabf9cc4455ad698f5158e71d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129ddeabf9cc4455ad698f5158e71d90" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35269 } } }
I20260929 18:38:13.258289 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:13.259677 13615 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:38:13.260432 13623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:13.266278 13623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:13.266983 13623 catalog_manager.cc:1259] Loaded cluster ID: d357c53b99fc4c9482f4e7037fdc554c
I20260929 18:38:13.267045 13623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:13.269654 13623 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:13.270484 13623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Loaded TSK: 0
I20260929 18:38:13.271827 13623 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:38:13.363283 13617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:13.363663 13617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:13.363732 13617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:13.369664 13617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:13.369751 13617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:13.369838 13617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:38:13.377669 13617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:39853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=37173
--tserver_master_addrs=127.0.198.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:13.379032 13617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:13.380561 13617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:13.394636 13652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:13.395094 13653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:13.395726 13655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:13.405961 13617 server_base.cc:1034] running on GCE node
I20260929 18:38:13.406605 13617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:13.407508 13617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:13.409090 13617 hybrid_clock.cc:648] HybridClock initialized: now 1790707093409038 us; error 63 us; skew 500 ppm
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:13.413017 13617 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:38:13.414326 13617 webserver.cc:460] Webserver started at http://127.0.198.1:37173/ using document root <none> and password file <none>
I20260929 18:38:13.415148 13617 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:13.415259 13617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:13.449429 13617 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.003s
I20260929 18:38:13.461007 13663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.462867 13617 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.002s
I20260929 18:38:13.463030 13617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "ff74c04badfa485c86b0832c7676de23"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "3fd5e132edc640a241da6ab186d90257"
server_key_iv: "86ef29a23fbd70be7cae54fe28f2ac5c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:13.463845 13617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:13.508636 13617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:13.512362 13617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:13.512579 13617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:13.513311 13617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:13.514398 13617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:13.514464 13617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.514531 13617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:13.514564 13617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.525867 13617 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:39853
I20260929 18:38:13.526412 13776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:39853 every 8 connection(s)
I20260929 18:38:13.526976 13617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:13.527704 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13617
I20260929 18:38:13.528141 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:45469
--local_ip_for_outbound_sockets=127.0.198.2
--tserver_master_addrs=127.0.198.62:35269
--webserver_port=43871
--webserver_interface=127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:13.550478 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:13.529632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35269 (local address 127.0.198.1:49763)
0929 18:38:13.530248 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:13.530314 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:13.531344 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:13.535695 (+ 4351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:13.535704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:13.536046 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:13.536749 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.536773 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.538801 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.538804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.540177 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.540188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.541420 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.541424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.541620 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.542329 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:13.542350 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:13.544021 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:13.546938 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:13.546945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:13.546960 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:13.547296 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:13.547681 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:13.547684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:13.547686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:13.547848 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:13.548605 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:13.548611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:13.548917 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 18:38:13.549107 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 18:38:13.549443 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":167,"threads_started":1}
I20260929 18:38:13.552212 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:13.530532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:49763 (local address 127.0.198.62:35269)
0929 18:38:13.534280 (+ 3748us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:13.534305 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:13.534366 (+ 61us) server_negotiation.cc:407] Connection header received
0929 18:38:13.534697 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:13.534715 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:13.535175 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:13.535570 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:13.536968 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.538649 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.540316 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.541042 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.541198 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.544247 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:13.544254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:13.544273 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:13.544305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:13.546805 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:13.547433 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:13.547440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:13.547445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:13.547575 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:13.547942 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:13.547945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:13.547947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:13.548433 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:13.548544 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:13.550988 (+ 2444us) server_negotiation.cc:299] Negotiation successful
0929 18:38:13.551268 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3572,"thread_start_us":111,"threads_started":1}
I20260929 18:38:13.555004 13777 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35269
I20260929 18:38:13.555269 13777 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:13.555853 13777 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
W20260929 18:38:13.558221 13777 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:35269 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:38:13.558341 13777 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:13.558525 13777 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
I20260929 18:38:13.559337 13777 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:13.559515 13777 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
W20260929 18:38:13.560259 13777 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:38:13.663695 13779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:13.664091 13779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:13.664156 13779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:13.670271 13779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:13.670363 13779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:13.670451 13779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:38:13.678421 13779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:45469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=43871
--tserver_master_addrs=127.0.198.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:13.679831 13779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:13.681222 13779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:13.693806 13786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:13.693806 13789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:13.694659 13779 server_base.cc:1034] running on GCE node
W20260929 18:38:13.695313 13787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:13.695843 13779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:13.696666 13779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:13.697849 13779 hybrid_clock.cc:648] HybridClock initialized: now 1790707093697815 us; error 50 us; skew 500 ppm
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:13.701079 13779 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:38:13.702077 13779 webserver.cc:460] Webserver started at http://127.0.198.2:43871/ using document root <none> and password file <none>
I20260929 18:38:13.702770 13779 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:13.702872 13779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:13.728813 13779 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260929 18:38:13.732296 13797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.733592 13779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:38:13.733709 13779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "0ab2b09c5e78465ea75a4b1cb3df5474"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "dea55dd3c42d3d01b6fdac4b07cb020a"
server_key_iv: "c22b1301b640e7f2e395ae94fc25046c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:13.734319 13779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:13.754673 13779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:13.757934 13779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:13.758149 13779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:13.758831 13779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:13.759732 13779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:13.759780 13779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.759832 13779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:13.759860 13779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.769701 13779 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:45469
I20260929 18:38:13.770865 13779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:13.772418 13911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:45469 every 8 connection(s)
I20260929 18:38:13.775534 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13779
I20260929 18:38:13.776023 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.198.20:40607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:45397
--local_ip_for_outbound_sockets=127.0.198.3
--tserver_master_addrs=127.0.198.62:35269
--webserver_port=38543
--webserver_interface=127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:13.786181 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:13.773029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:43087 (local address 127.0.198.62:35269)
0929 18:38:13.773153 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:13.773157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:13.774301 (+ 1144us) server_negotiation.cc:407] Connection header received
0929 18:38:13.775300 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:13.775303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:13.775364 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:13.775444 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:13.776636 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.777165 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.778482 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.778833 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.778965 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.781844 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:13.781849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:13.781852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:13.781886 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:13.783660 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:13.784333 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:13.784337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:13.784339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:13.784436 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:13.784836 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:13.784839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:13.784841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:13.785057 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:13.785146 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:13.785842 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 18:38:13.786009 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:38:13.787603 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:13.773563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35269 (local address 127.0.198.2:43087)
0929 18:38:13.774144 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:13.774203 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:13.775098 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:13.775628 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:13.775638 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:13.775921 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:13.776489 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.776505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.777287 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.777289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.778340 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.778352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.779028 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.779033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.779236 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.779863 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:13.779882 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:13.781621 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:13.783801 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:13.783814 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:13.783835 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:13.784220 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:13.784531 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:13.784534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:13.784536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:13.784744 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:13.785160 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:13.785168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:13.785572 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 18:38:13.785813 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 18:38:13.786206 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":142,"threads_started":1}
I20260929 18:38:13.789148 13912 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35269
I20260929 18:38:13.789469 13912 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:13.790102 13912 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
W20260929 18:38:13.791956 13912 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:35269 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:38:13.792135 13912 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:13.792332 13912 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
I20260929 18:38:13.793182 13912 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:13.793385 13912 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
W20260929 18:38:13.794137 13912 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:38:13.911965 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:13.912379 13915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:13.912458 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:13.918624 13915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:13.918715 13915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:13.918802 13915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:38:13.925259 13915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.198.20:40607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:45397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=38543
--tserver_master_addrs=127.0.198.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:13.926445 13915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:13.927498 13915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:13.939198 13923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:13.939198 13920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:13.939656 13921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:13.942008 13915 server_base.cc:1034] running on GCE node
I20260929 18:38:13.942500 13915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:13.943329 13915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:13.944536 13915 hybrid_clock.cc:648] HybridClock initialized: now 1790707093944465 us; error 91 us; skew 500 ppm
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:13.948215 13915 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:38:13.949404 13915 webserver.cc:460] Webserver started at http://127.0.198.3:38543/ using document root <none> and password file <none>
I20260929 18:38:13.950148 13915 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:13.950253 13915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:38:13 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:13.978709 13915 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260929 18:38:13.982009 13931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:13.983707 13915 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:38:13.983846 13915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb"
format_stamp: "Formatted at 2026-09-29 18:38:12 on dist-test-slave-q5qx"
server_key: "625b4bacab2002384ad060cdd68aa99f"
server_key_iv: "ec1bdbbb84ba3c79dbc988fce713ef1c"
server_key_version: "kuduclusterkey@0"
I20260929 18:38:13.984561 13915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:14.006951 13915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:14.010617 13915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:14.010824 13915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:14.011544 13915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:14.012573 13915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:14.012636 13915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:14.012689 13915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:14.012717 13915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:14.024179 13915 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:45397
I20260929 18:38:14.024263 14051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:45397 every 8 connection(s)
I20260929 18:38:14.025534 13915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:14.035084 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 13915
Sep 29 18:38:14 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:14.040117 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.027532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:59809 (local address 127.0.198.62:35269)
0929 18:38:14.027682 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.027687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.028603 (+ 916us) server_negotiation.cc:407] Connection header received
0929 18:38:14.029564 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.029568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.029627 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.029720 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:14.031021 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.031812 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.033010 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.033356 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.033482 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.036055 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:14.036059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:14.036063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:14.036091 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:14.037853 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:14.038431 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:14.038434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:14.038436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:14.038511 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:14.038882 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:14.038885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:14.038887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:14.039062 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:14.039190 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.039778 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.039946 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:38:14.040946 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.027944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:35269 (local address 127.0.198.3:59809)
0929 18:38:14.028423 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.028477 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.029341 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.030089 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.030097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.030367 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:14.030866 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.030884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.031949 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.031953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.032887 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.032896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.033507 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.033509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.033684 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.034277 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:14.034293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:14.035855 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:14.037965 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:14.037994 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:14.038012 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:14.038300 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:14.038592 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:14.038595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:14.038597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:14.038729 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:14.039196 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:14.039203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:14.039522 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 18:38:14.039736 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 18:38:14.040053 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260929 18:38:14.042140 14052 heartbeater.cc:347] Connected to a master server at 127.0.198.62:35269
I20260929 18:38:14.042413 14052 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:14.043016 14052 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
I20260929 18:38:14.044497 13579 ts_manager.cc:194] Registered new tserver with Master: 5d1a26466c7644f1b5ac7e6e21e9ddeb (127.0.198.3:45397)
I20260929 18:38:14.045284 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.036025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.0.198.62:35269)
0929 18:38:14.036301 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.036307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.036318 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:38:14.036347 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.036349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.036389 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.037281 (+ 892us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:14.037361 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.037921 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.039224 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.039525 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.039623 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.041531 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:14.041535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:14.041537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:14.041562 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:14.043080 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:14.043962 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:14.043965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:14.043967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:14.044102 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:14.044431 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:14.044434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:14.044435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:14.044638 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:14.044789 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.045004 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.045128 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":118,"threads_started":1}
I20260929 18:38:14.045789 13579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:59809
Sep 29 18:38:14 dist-test-slave-q5qx krb5kdc[12583](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:14.560487 13777 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:14.560793 13777 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
I20260929 18:38:14.561545 13579 ts_manager.cc:194] Registered new tserver with Master: ff74c04badfa485c86b0832c7676de23 (127.0.198.1:39853)
I20260929 18:38:14.562213 13579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:49763
Sep 29 18:38:14 dist-test-slave-q5qx krb5kdc[12583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for HTTP/127.0.198.20@KRBTEST.COM
I20260929 18:38:14.794405 13912 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:14.794698 13912 heartbeater.cc:511] Master 127.0.198.62:35269 requested a full tablet report, sending...
I20260929 18:38:14.795514 13579 ts_manager.cc:194] Registered new tserver with Master: 0ab2b09c5e78465ea75a4b1cb3df5474 (127.0.198.2:45469)
I20260929 18:38:14.796147 13579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:43087
I20260929 18:38:14.802549 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:38:14.813783 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.804493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.0.198.62:35269)
0929 18:38:14.805126 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.805130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.805142 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:14.805180 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.805183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.805229 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.805316 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:14.806176 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.807070 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.807973 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.808305 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.808396 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.809938 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:14.809942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:14.809945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:14.809998 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:14.811629 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:14.812350 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:14.812355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:14.812357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:14.812482 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:14.812773 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:14.812776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:14.812778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:14.813008 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:14.813137 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.813521 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.813649 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":91,"threads_started":1}
I20260929 18:38:14.817713 13579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:38:14.821414 13579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:38:14.844872 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.837153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:45397 (local address 127.0.0.1:43428)
0929 18:38:14.837824 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.837877 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.838037 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.839750 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.839757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.839784 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:14.839949 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.839958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.841637 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.841641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.843096 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.843108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.844314 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.844317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.844443 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.844491 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:38:14.844581 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:38:14.844708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":116,"threads_started":1}
I20260929 18:38:14.846555 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.837305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.0.198.3:45397)
0929 18:38:14.839183 (+ 1878us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.839190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.839218 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:38:14.839314 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.839320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.839562 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.840182 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:14.840327 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.841499 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.843256 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.844177 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.846083 (+ 1906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.846253 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.846309 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.846409 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1671,"thread_start_us":77,"threads_started":1}
I20260929 18:38:14.846731 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.838584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:45469 (local address 127.0.0.1:34726)
0929 18:38:14.840587 (+ 2003us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.840599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.840667 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.841308 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.841311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.841328 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:14.841504 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.841512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.843700 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.843703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.844806 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.844816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.846338 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.846341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.846439 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.846462 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:38:14.846528 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:38:14.846596 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1951,"thread_start_us":90,"threads_started":1}
I20260929 18:38:14.847188 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.838866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.0.198.2:45469)
0929 18:38:14.840070 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.840077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.840748 (+ 671us) server_negotiation.cc:407] Connection header received
0929 18:38:14.840825 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.840830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.841034 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.841166 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:14.841630 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.843572 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.844939 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.846056 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.846149 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.846281 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.846957 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.847058 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":137,"threads_started":1}
I20260929 18:38:14.849947 13846 tablet_service.cc:1467] Processing CreateTablet for tablet a975ba87afe8459b94796997c28f0e29 (DEFAULT_TABLE table=test-table [id=1b322d69a11944c3a4800e8b0757173a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:14.849947 13978 tablet_service.cc:1467] Processing CreateTablet for tablet a975ba87afe8459b94796997c28f0e29 (DEFAULT_TABLE table=test-table [id=1b322d69a11944c3a4800e8b0757173a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:14.851059 13846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a975ba87afe8459b94796997c28f0e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:14.851074 13978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a975ba87afe8459b94796997c28f0e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:14.854982 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.838347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:39853 (local address 127.0.0.1:41164)
0929 18:38:14.840309 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.840323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.840411 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.850861 (+ 10450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.850864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.850885 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:14.850976 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.850982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.852307 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.852311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.853412 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.853423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.854569 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.854576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.854667 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.854696 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:38:14.854749 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:38:14.854826 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"thread_start_us":107,"threads_started":1}
I20260929 18:38:14.855758 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.838508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.0.198.1:39853)
0929 18:38:14.850203 (+ 11695us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.850210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.850235 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:38:14.850308 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.850313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.850515 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.850689 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:14.851110 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.852176 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.853582 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.854400 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.855268 (+ 868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.855421 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.855471 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.855578 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11579,"thread_start_us":111,"threads_started":1}
I20260929 18:38:14.858848 13708 tablet_service.cc:1467] Processing CreateTablet for tablet a975ba87afe8459b94796997c28f0e29 (DEFAULT_TABLE table=test-table [id=1b322d69a11944c3a4800e8b0757173a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:14.859968 13708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a975ba87afe8459b94796997c28f0e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:14.865813 14079 tablet_bootstrap.cc:492] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: Bootstrap starting.
I20260929 18:38:14.868132 14079 tablet_bootstrap.cc:654] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:14.869026 14079 log.cc:824] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:14.869791 14080 tablet_bootstrap.cc:492] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23: Bootstrap starting.
I20260929 18:38:14.870123 14078 tablet_bootstrap.cc:492] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Bootstrap starting.
I20260929 18:38:14.872113 14080 tablet_bootstrap.cc:654] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:14.872126 14078 tablet_bootstrap.cc:654] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:14.872885 14078 log.cc:824] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:14.872889 14080 log.cc:824] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:14.875203 14079 tablet_bootstrap.cc:492] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: No bootstrap required, opened a new log
I20260929 18:38:14.875406 14079 ts_tablet_manager.cc:1397] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 18:38:14.876076 14080 tablet_bootstrap.cc:492] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23: No bootstrap required, opened a new log
I20260929 18:38:14.876245 14080 ts_tablet_manager.cc:1397] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:38:14.879834 14080 raft_consensus.cc:348] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.880090 14080 raft_consensus.cc:374] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:14.880134 14080 raft_consensus.cc:729] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff74c04badfa485c86b0832c7676de23, State: Initialized, Role: FOLLOWER
I20260929 18:38:14.880702 14080 consensus_queue.cc:260] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.879797 14079 raft_consensus.cc:348] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.881506 14079 raft_consensus.cc:374] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:14.881625 14079 raft_consensus.cc:729] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab2b09c5e78465ea75a4b1cb3df5474, State: Initialized, Role: FOLLOWER
I20260929 18:38:14.881804 14080 ts_tablet_manager.cc:1428] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:38:14.882220 14079 consensus_queue.cc:260] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.883280 14079 ts_tablet_manager.cc:1428] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:38:14.883642 14078 tablet_bootstrap.cc:492] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: No bootstrap required, opened a new log
I20260929 18:38:14.883690 13912 heartbeater.cc:503] Master 127.0.198.62:35269 was elected leader, sending a full tablet report...
I20260929 18:38:14.883865 14078 ts_tablet_manager.cc:1397] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260929 18:38:14.889074 14078 raft_consensus.cc:348] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.889354 14078 raft_consensus.cc:374] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:14.889411 14078 raft_consensus.cc:729] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1a26466c7644f1b5ac7e6e21e9ddeb, State: Initialized, Role: FOLLOWER
I20260929 18:38:14.889853 14078 consensus_queue.cc:260] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.890954 14078 ts_tablet_manager.cc:1428] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 18:38:14.891320 14052 heartbeater.cc:503] Master 127.0.198.62:35269 was elected leader, sending a full tablet report...
I20260929 18:38:14.891428 13777 heartbeater.cc:503] Master 127.0.198.62:35269 was elected leader, sending a full tablet report...
I20260929 18:38:14.922917 14084 raft_consensus.cc:482] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:14.923170 14084 raft_consensus.cc:504] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.924571 14084 leader_election.cc:288] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d1a26466c7644f1b5ac7e6e21e9ddeb (127.0.198.3:45397), 0ab2b09c5e78465ea75a4b1cb3df5474 (127.0.198.2:45469)
I20260929 18:38:14.929915 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.924961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:44299 (local address 127.0.198.3:45397)
0929 18:38:14.925087 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.925092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.925358 (+ 266us) server_negotiation.cc:407] Connection header received
0929 18:38:14.925583 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.925587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.925641 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.925688 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:14.926161 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.926939 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.928585 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.929208 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.929284 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.929350 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.929697 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.929779 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:38:14.930295 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.925608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:43217 (local address 127.0.198.2:45469)
0929 18:38:14.925720 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.925724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.926015 (+ 291us) server_negotiation.cc:407] Connection header received
0929 18:38:14.926230 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.926234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.926287 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.926332 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:14.926727 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.927516 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.929010 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.929612 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.929690 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.929751 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.930099 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.930179 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:38:14.930781 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.925242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:45469 (local address 127.0.198.1:43217)
0929 18:38:14.925894 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.925908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.926098 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.926457 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.926460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.926477 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:14.926594 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.926601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.927655 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.927658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.928861 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.928873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.929840 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.929843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.929940 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.929960 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:38:14.930541 (+ 581us) client_negotiation.cc:240] Negotiation successful
0929 18:38:14.930618 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":16,"thread_start_us":93,"threads_started":1}
I20260929 18:38:14.930816 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.924819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:45397 (local address 127.0.198.1:44299)
0929 18:38:14.925261 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.925275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.925453 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.925834 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.925837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.925855 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:14.926026 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.926035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.927080 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.927084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.928446 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.928457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.929456 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.929460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.929560 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.929583 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:38:14.930582 (+ 999us) client_negotiation.cc:240] Negotiation successful
0929 18:38:14.930664 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":39,"thread_start_us":128,"threads_started":1}
I20260929 18:38:14.931427 13866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a975ba87afe8459b94796997c28f0e29" candidate_uuid: "ff74c04badfa485c86b0832c7676de23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" is_pre_election: true
I20260929 18:38:14.931768 13866 raft_consensus.cc:2454] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff74c04badfa485c86b0832c7676de23 in term 0.
I20260929 18:38:14.932936 13666 leader_election.cc:302] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab2b09c5e78465ea75a4b1cb3df5474, ff74c04badfa485c86b0832c7676de23; no voters:
I20260929 18:38:14.933286 14084 raft_consensus.cc:2781] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:38:14.933382 14084 raft_consensus.cc:482] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:38:14.933429 14084 raft_consensus.cc:3037] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:14.933544 14005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a975ba87afe8459b94796997c28f0e29" candidate_uuid: "ff74c04badfa485c86b0832c7676de23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" is_pre_election: true
I20260929 18:38:14.933841 14005 raft_consensus.cc:2454] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff74c04badfa485c86b0832c7676de23 in term 0.
I20260929 18:38:14.934725 14084 raft_consensus.cc:504] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.935079 14084 leader_election.cc:288] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [CANDIDATE]: Term 1 election: Requested vote from peers 5d1a26466c7644f1b5ac7e6e21e9ddeb (127.0.198.3:45397), 0ab2b09c5e78465ea75a4b1cb3df5474 (127.0.198.2:45469)
I20260929 18:38:14.935632 14005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a975ba87afe8459b94796997c28f0e29" candidate_uuid: "ff74c04badfa485c86b0832c7676de23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb"
I20260929 18:38:14.935757 14005 raft_consensus.cc:3037] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:14.937084 14005 raft_consensus.cc:2454] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff74c04badfa485c86b0832c7676de23 in term 1.
I20260929 18:38:14.937422 13666 leader_election.cc:302] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1a26466c7644f1b5ac7e6e21e9ddeb, ff74c04badfa485c86b0832c7676de23; no voters:
I20260929 18:38:14.937677 13866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a975ba87afe8459b94796997c28f0e29" candidate_uuid: "ff74c04badfa485c86b0832c7676de23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474"
I20260929 18:38:14.937806 13866 raft_consensus.cc:3037] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:14.938050 14084 raft_consensus.cc:2781] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:14.938302 14084 raft_consensus.cc:686] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [term 1 LEADER]: Becoming Leader. State: Replica: ff74c04badfa485c86b0832c7676de23, State: Running, Role: LEADER
I20260929 18:38:14.938660 14084 consensus_queue.cc:237] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } }
I20260929 18:38:14.939190 13866 raft_consensus.cc:2454] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff74c04badfa485c86b0832c7676de23 in term 1.
I20260929 18:38:14.941951 13579 catalog_manager.cc:5495] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff74c04badfa485c86b0832c7676de23 (127.0.198.1). New cstate: current_term: 1 leader_uuid: "ff74c04badfa485c86b0832c7676de23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff74c04badfa485c86b0832c7676de23" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 39853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 } health_report { overall_health: UNKNOWN } } }
I20260929 18:38:14.974596 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.969810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.0.198.1:39853)
0929 18:38:14.970134 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.970139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.970155 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:38:14.970245 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.970248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.970319 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.970437 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:38:14.970690 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.971501 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.972637 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.972969 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.973089 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.973444 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:38:14.974115 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:38:14.974245 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:14.974354 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:38:14.974421 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260929 18:38:14.981778 13866 raft_consensus.cc:1261] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 1 FOLLOWER]: Refusing update from remote peer ff74c04badfa485c86b0832c7676de23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:14.981860 14005 raft_consensus.cc:1261] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 1 FOLLOWER]: Refusing update from remote peer ff74c04badfa485c86b0832c7676de23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:14.982785 14089 consensus_queue.cc:1035] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab2b09c5e78465ea75a4b1cb3df5474" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 45469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:14.982946 14084 consensus_queue.cc:1035] T a975ba87afe8459b94796997c28f0e29 P ff74c04badfa485c86b0832c7676de23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1a26466c7644f1b5ac7e6e21e9ddeb" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 45397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:14.993934 14092 mvcc.cc:205] Tried to move back new op lower bound from 7334736261037502464 to 7334736260871622656. Current Snapshot: MvccSnapshot[applied={T|T < 7334736261037502464}]
I20260929 18:38:15.006086 14095 mvcc.cc:205] Tried to move back new op lower bound from 7334736261037502464 to 7334736260871622656. Current Snapshot: MvccSnapshot[applied={T|T < 7334736261037502464}]
I20260929 18:38:15.007155 13579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:15.007339 13579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:15.010366 13579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 129ddeabf9cc4455ad698f5158e71d90: Sending DeleteTablet for 3 replicas of tablet a975ba87afe8459b94796997c28f0e29
I20260929 18:38:15.011389 13978 tablet_service.cc:1514] Processing DeleteTablet for tablet a975ba87afe8459b94796997c28f0e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:15 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:43428
I20260929 18:38:15.011420 13708 tablet_service.cc:1514] Processing DeleteTablet for tablet a975ba87afe8459b94796997c28f0e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:15 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:41164
I20260929 18:38:15.011648 13846 tablet_service.cc:1514] Processing DeleteTablet for tablet a975ba87afe8459b94796997c28f0e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:15 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:34726
I20260929 18:38:15.013142 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13617
I20260929 18:38:15.015934 14103 tablet_replica.cc:330] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: stopping tablet replica
I20260929 18:38:15.016256 14103 raft_consensus.cc:2229] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:15.016589 14103 raft_consensus.cc:2258] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:15.017565 14102 tablet_replica.cc:330] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: stopping tablet replica
I20260929 18:38:15.017894 14102 raft_consensus.cc:2229] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:15.018255 14102 raft_consensus.cc:2258] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:15.019976 14102 ts_tablet_manager.cc:1905] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:38:15.021255 13566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:39853 (error 108)
I20260929 18:38:15.021330 14103 ts_tablet_manager.cc:1905] T a975ba87afe8459b94796997c28f0e29 P 0ab2b09c5e78465ea75a4b1cb3df5474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:15.022655 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13779
I20260929 18:38:15.024010 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:15.022390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:39853 (local address 127.0.0.1:41190)
0929 18:38:15.022500 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:15.022571 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 18:38:15.024201 13566 catalog_manager.cc:4584] TS ff74c04badfa485c86b0832c7676de23 (127.0.198.1:39853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a975ba87afe8459b94796997c28f0e29: Network error: Client connection negotiation failed: client connection to 127.0.198.1:39853: connect: Connection refused (error 111)
I20260929 18:38:15.027324 14102 ts_tablet_manager.cc:1918] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:15.027431 14102 log.cc:1196] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/a975ba87afe8459b94796997c28f0e29
I20260929 18:38:15.027732 14102 ts_tablet_manager.cc:1939] T a975ba87afe8459b94796997c28f0e29 P 5d1a26466c7644f1b5ac7e6e21e9ddeb: Deleting consensus metadata
I20260929 18:38:15.029101 13565 catalog_manager.cc:4841] TS 5d1a26466c7644f1b5ac7e6e21e9ddeb (127.0.198.3:45397): tablet a975ba87afe8459b94796997c28f0e29 (table test-table [id=1b322d69a11944c3a4800e8b0757173a]) successfully deleted
W20260929 18:38:15.029793 14074 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:15.029390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:45469 (local address 127.0.0.1:34730)
0929 18:38:15.029528 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:15.029543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:15.029620 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:15.029703 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.2:45469: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:38:15.029963 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13915
I20260929 18:38:15.035470 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 13546
I20260929 18:38:15.041072 792 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:38:15.376088 792 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:38:15.376187 792 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:38:15.711150 792 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:38:15.711 UTC [12613] LOG: received smart shutdown request
2026-09-29 18:38:15.715 UTC [12613] LOG: background worker "logical replication launcher" (PID 12622) exited with exit code 1
2026-09-29 18:38:15.715 UTC [12617] LOG: shutting down
2026-09-29 18:38:15.887 UTC [12613] LOG: database system is shut down
2026-09-29T18:38:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47948 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:38:16 dist-test-slave-q5qx krb5kdc[14113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:38:16 dist-test-slave-q5qx krb5kdc[14113](info): set up 2 sockets
Sep 29 18:38:16 dist-test-slave-q5qx krb5kdc[14113](info): commencing operation
krb5kdc: starting...
W20260929 18:38:16.945822 792 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:38:17.065649 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39355
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39355,127.0.198.61:33129,127.0.198.60:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:17.174331 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:17.174618 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:17.174678 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:17.178666 14126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:17.178725 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:17.178751 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:17.178771 14126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:17.178791 14126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:17.184099 14126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39355,127.0.198.61:33129,127.0.198.60:44185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:17.185141 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:17.186200 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:17.194279 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.194327 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.194309 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.194675 14126 server_base.cc:1034] running on GCE node
I20260929 18:38:17.195221 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:17.196292 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:17.197451 14126 hybrid_clock.cc:648] HybridClock initialized: now 1790707097197441 us; error 30 us; skew 500 ppm
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:17.200173 14126 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:17.201020 14126 webserver.cc:460] Webserver started at http://127.0.198.62:46201/ using document root <none> and password file <none>
I20260929 18:38:17.201527 14126 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:17.201582 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:17.201738 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.203833 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "26028a3fce954df3a0c0a06cad15ffc4"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "72db1fc61fe63c0b0b78a0f3a0ba2bf4"
server_key_iv: "5e5cc455541fb6b0373aafc2689d06fc"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.204365 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "26028a3fce954df3a0c0a06cad15ffc4"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "72db1fc61fe63c0b0b78a0f3a0ba2bf4"
server_key_iv: "5e5cc455541fb6b0373aafc2689d06fc"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.208133 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 18:38:17.210654 14141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.211900 14126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:17.212011 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "26028a3fce954df3a0c0a06cad15ffc4"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "72db1fc61fe63c0b0b78a0f3a0ba2bf4"
server_key_iv: "5e5cc455541fb6b0373aafc2689d06fc"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.212117 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.250402 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.253358 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.253537 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.260035 14126 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:39355
I20260929 18:38:17.260059 14193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:39355 every 8 connection(s)
I20260929 18:38:17.260946 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:17.261857 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14126
I20260929 18:38:17.262001 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:38:17.262306 792 external_mini_cluster.cc:1428] Setting key 58f135ec35cc162121528ad98a9001de
I20260929 18:38:17.264281 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:17.268186 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.61@KRBTEST.COM" created.
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:17.283495 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.270410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39355 (local address 127.0.0.1:40072)
0929 18:38:17.270882 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.270923 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.271808 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.272481 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.272487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.272935 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.273202 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.273215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.274832 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.274835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.275681 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.275690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.276559 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.276562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.276722 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.277330 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.277345 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.278936 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.280774 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.280792 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.280805 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.281046 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.281365 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.281368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.281369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.281499 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.282079 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.282085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.282226 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.282370 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.282760 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":172,"thread_start_us":126,"threads_started":1}
I20260929 18:38:17.283522 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.271198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.0.198.62:39355)
0929 18:38:17.271473 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.271496 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.271539 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:38:17.272112 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.272117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.272300 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.272444 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.273338 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.274692 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.275810 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.276396 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.276623 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.279088 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.279093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.279104 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.279126 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.280663 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.281147 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.281151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.281152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.281253 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.281591 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.281594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.281595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.281953 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.282102 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.282439 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.282627 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260929 18:38:17.285776 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:38:17.286440 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.286217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33042)
0929 18:38:17.286320 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.286386 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:38:17.286644 14145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
I20260929 18:38:17.287456 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.287273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33058)
0929 18:38:17.287367 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.287407 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:38:17.287609 14194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:33129: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab.
Entry for principal kudu/127.0.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab.
I20260929 18:38:17.288691 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:33129
--webserver_interface=127.0.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab
--principal=kudu/127.0.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39355,127.0.198.61:33129,127.0.198.60:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:17.336848 14194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } attempt: 1
I20260929 18:38:17.337765 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.337429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33060)
0929 18:38:17.337624 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.337688 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 18:38:17.338614 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.338386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33072)
0929 18:38:17.338512 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.338556 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:38:17.338747 14194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:33129: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
W20260929 18:38:17.398080 14201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:17.398350 14201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:17.398408 14201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:17.402483 14201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:17.402554 14201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:17.402580 14201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:17.402602 14201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:17.402630 14201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:17.406899 14194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } attempt: 2
I20260929 18:38:17.407706 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.407380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33076)
0929 18:38:17.407570 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.407631 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:38:17.408248 14201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39355,127.0.198.61:33129,127.0.198.60:44185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab
--principal=kudu/127.0.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:33129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:17.408515 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.408262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33088)
0929 18:38:17.408382 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.408444 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:38:17.408627 14194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:33129: Network error: Client connection negotiation failed: client connection to 127.0.198.61:33129: connect: Connection refused (error 111)
I20260929 18:38:17.409384 14201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:17.410421 14201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:17.418727 14206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.418742 14209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.418727 14207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.419106 14201 server_base.cc:1034] running on GCE node
I20260929 18:38:17.419721 14201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:17.420820 14201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:17.422019 14201 hybrid_clock.cc:648] HybridClock initialized: now 1790707097421997 us; error 31 us; skew 500 ppm
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:17.424994 14201 init.cc:377] Logged in from keytab as kudu/127.0.198.61@KRBTEST.COM (short username kudu)
I20260929 18:38:17.425918 14201 webserver.cc:460] Webserver started at http://127.0.198.61:36165/ using document root <none> and password file <none>
I20260929 18:38:17.426489 14201 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:17.426556 14201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:17.426750 14201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.428915 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data/instance:
uuid: "ddc1d964af2e49d3a3825666f7fd083f"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "d3831cb83e208bfaef28dab878a4f237"
server_key_iv: "bc783488ef78a97189a09162bed33a88"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.429494 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/wal/instance:
uuid: "ddc1d964af2e49d3a3825666f7fd083f"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "d3831cb83e208bfaef28dab878a4f237"
server_key_iv: "bc783488ef78a97189a09162bed33a88"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.433465 14201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:38:17.435933 14216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.437227 14201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:17.437341 14201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/wal
uuid: "ddc1d964af2e49d3a3825666f7fd083f"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "d3831cb83e208bfaef28dab878a4f237"
server_key_iv: "bc783488ef78a97189a09162bed33a88"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.437434 14201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.450848 14201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.453766 14201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.453939 14201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.461138 14201 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.61:33129
I20260929 18:38:17.461190 14268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.61:33129 every 8 connection(s)
I20260929 18:38:17.462141 14201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
I20260929 18:38:17.464557 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14201
I20260929 18:38:17.464671 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-1/wal/instance
I20260929 18:38:17.464990 792 external_mini_cluster.cc:1428] Setting key f9a93692140aa1d0c502f092528ed81d
I20260929 18:38:17.465158 14269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:17.469256 14269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.198.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.60@KRBTEST.COM" created.
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:17.482980 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.471193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.0.198.62:39355)
0929 18:38:17.471346 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.471351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.472209 (+ 858us) server_negotiation.cc:407] Connection header received
0929 18:38:17.473125 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.473128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.473178 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.473259 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.474510 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.475146 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.476267 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.476633 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.476741 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.479328 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.479331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.479333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.479360 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.480779 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.481368 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.481371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.481372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.481453 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.481810 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.481812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.481814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.482031 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.482109 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.482738 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.482845 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:38:17.483814 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.471557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39355 (local address 127.0.0.1:40080)
0929 18:38:17.472036 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.472088 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.472936 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.473498 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.473505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.473833 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.474357 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.474380 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.475277 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.475280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.476141 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.476151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.476764 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.476767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.476903 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.477543 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.477560 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.479112 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.480900 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.480906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.480922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.481260 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.481551 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.481554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.481556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.481728 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.482153 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.482158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.482491 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.482692 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.482977 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260929 18:38:17.485143 14269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab.
Entry for principal kudu/127.0.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab.
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
I20260929 18:38:17.490777 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:44185
--webserver_interface=127.0.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab
--principal=kudu/127.0.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39355,127.0.198.61:33129,127.0.198.60:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:17.493938 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.485622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33092)
0929 18:38:17.485716 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.485729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.485802 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.486333 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.486336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.486533 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.486636 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.486642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.487630 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.487635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.488237 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.488246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.489037 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.489040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.489122 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.489550 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.489561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.490732 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.492467 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.492471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.492473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.492710 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.492988 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.492991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.492992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.493061 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.493536 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.493539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.493624 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.493716 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.493815 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:38:17.494170 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.485683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.0.198.61:33129)
0929 18:38:17.485948 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.485957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.486002 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:38:17.486045 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.486050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.486214 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.486316 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.486734 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.487511 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.488329 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.488899 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.488999 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.490852 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.490863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.490880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.490906 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.492360 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.492819 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.492825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.492827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.492910 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.493134 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.493137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.493138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.493415 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.493540 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.493814 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.493956 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
I20260929 18:38:17.496277 14269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:38:17.496977 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.496729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57128)
0929 18:38:17.496838 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.496922 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:38:17.497184 14219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
I20260929 18:38:17.498060 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.497818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57140)
0929 18:38:17.497927 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.497992 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:38:17.498195 14269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:44185: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
I20260929 18:38:17.499778 14194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } attempt: 3
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
I20260929 18:38:17.509080 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.500210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33108)
0929 18:38:17.500330 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.500341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.500421 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.500611 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.500612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.500751 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.501183 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.501190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.502630 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.502633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.503206 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.503216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.503805 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.503809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.503912 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.504404 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.504416 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.505872 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.507661 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.507664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.507665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.507853 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.508130 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.508132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.508134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.508224 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.508623 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.508625 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.508732 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.508823 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.508954 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 18:38:17.509153 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.500234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.0.198.61:33129)
0929 18:38:17.500390 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.500395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.500409 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:38:17.500483 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.500487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.500527 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.500595 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.501279 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.502507 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.503307 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.503683 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.503777 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.506031 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.506035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.506038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.506062 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.507545 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.507935 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.507938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.507940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.508025 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.508305 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.508307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.508309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.508508 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.508603 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.508880 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.508990 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 18:38:17.510151 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:38:17.510847 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.510623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57144)
0929 18:38:17.510738 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.510787 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:38:17.511471 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.511244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57158)
0929 18:38:17.511367 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.511419 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 18:38:17.511572 14194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:44185: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
I20260929 18:38:17.542724 14194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } attempt: 1
I20260929 18:38:17.543624 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.543284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57166)
0929 18:38:17.543478 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.543542 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 18:38:17.544312 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.544107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57172)
0929 18:38:17.544209 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.544254 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:38:17.544426 14194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:44185: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
I20260929 18:38:17.547444 14269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } attempt: 1
I20260929 18:38:17.548143 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.547890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57182)
0929 18:38:17.548022 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.548075 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:38:17.548811 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.548611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57196)
0929 18:38:17.548717 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.548759 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:38:17.548918 14269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:44185: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
W20260929 18:38:17.602947 14275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:17.603220 14275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:17.603278 14275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:17.607245 14275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:17.607298 14275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:17.607324 14275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:17.607347 14275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:17.607379 14275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:17.612627 14275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39355,127.0.198.61:33129,127.0.198.60:44185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab
--principal=kudu/127.0.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:44185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:17.613662 14275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:17.614699 14275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:17.617064 14269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } attempt: 2
I20260929 18:38:17.617801 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.617534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57208)
0929 18:38:17.617671 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.617729 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:38:17.618534 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.618277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57212)
0929 18:38:17.618379 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.618422 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:38:17.618692 14269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:44185: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
I20260929 18:38:17.619587 14194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } attempt: 2
I20260929 18:38:17.620194 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.619958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57228)
0929 18:38:17.620080 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.620130 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:38:17.620863 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.620684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57234)
0929 18:38:17.620773 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.620814 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 18:38:17.620967 14194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:44185: Network error: Client connection negotiation failed: client connection to 127.0.198.60:44185: connect: Connection refused (error 111)
W20260929 18:38:17.623463 14282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.623509 14281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.623468 14284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.623663 14275 server_base.cc:1034] running on GCE node
I20260929 18:38:17.624193 14275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:17.625284 14275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:17.626451 14275 hybrid_clock.cc:648] HybridClock initialized: now 1790707097626433 us; error 38 us; skew 500 ppm
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:17.629225 14275 init.cc:377] Logged in from keytab as kudu/127.0.198.60@KRBTEST.COM (short username kudu)
I20260929 18:38:17.630191 14275 webserver.cc:460] Webserver started at http://127.0.198.60:43699/ using document root <none> and password file <none>
I20260929 18:38:17.630723 14275 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:17.630782 14275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:17.630976 14275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.632982 14275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data/instance:
uuid: "40e0708f1e74426982cda6f2eb89e8ca"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "981f8271a7920632f30eeff52af29c50"
server_key_iv: "c09c15f72fd5e845626474d9262eadfd"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.633545 14275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/wal/instance:
uuid: "40e0708f1e74426982cda6f2eb89e8ca"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "981f8271a7920632f30eeff52af29c50"
server_key_iv: "c09c15f72fd5e845626474d9262eadfd"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.637313 14275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:38:17.639839 14291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.641196 14275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:38:17.641330 14275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/wal
uuid: "40e0708f1e74426982cda6f2eb89e8ca"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-q5qx"
server_key: "981f8271a7920632f30eeff52af29c50"
server_key_iv: "c09c15f72fd5e845626474d9262eadfd"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.641451 14275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.662909 14275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.665902 14275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.666115 14275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.673137 14275 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.60:44185
I20260929 18:38:17.673151 14343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.60:44185 every 8 connection(s)
I20260929 18:38:17.674152 14275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
I20260929 18:38:17.677038 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14275
I20260929 18:38:17.677165 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706942746699-792-0/minicluster-data/master-2/wal/instance
I20260929 18:38:17.677165 14344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:17.677469 792 external_mini_cluster.cc:1428] Setting key b235a85b8db82c18d924c5df00d8b67a
I20260929 18:38:17.682086 14344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:17.692200 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.678808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.198.60:44185)
0929 18:38:17.679205 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.679221 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.679257 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:38:17.679998 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.680019 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.680375 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.680723 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.681586 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.682691 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.683768 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.684298 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.684443 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.686904 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.686910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.686922 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.686952 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.689014 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.689595 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.689599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.689604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.689707 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.689993 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.689997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.689998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.690500 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.690718 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.690982 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.691256 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260929 18:38:17.692938 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.683865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39355 (local address 127.0.0.1:40092)
0929 18:38:17.684319 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.684366 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.684520 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.684841 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.684845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.685164 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.685281 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.685286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.686232 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.686235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.686937 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.686946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.687516 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.687521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.687638 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.688156 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.688170 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.689759 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.691517 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.691524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.691528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.691732 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.691984 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.691986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.691987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.692061 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.692476 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.692482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.692590 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.692698 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.692820 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260929 18:38:17.693289 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.683958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.0.198.62:39355)
0929 18:38:17.684102 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.684106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.684438 (+ 332us) server_negotiation.cc:407] Connection header received
0929 18:38:17.684621 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.684624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.684666 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.684706 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.685406 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.686124 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.687055 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.687325 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.687504 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.689885 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.689887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.689889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.689912 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.691422 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.691816 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.691819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.691821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.691903 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.692131 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.692134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.692136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.692341 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.692428 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.692800 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.692911 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:38:17.694146 14344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:17.703034 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.694578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:33129 (local address 127.0.0.1:33120)
0929 18:38:17.694801 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.694813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.694881 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.695121 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.695124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.695272 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.695380 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.695386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.696356 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.696359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.697198 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.697207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.697755 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.697758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.697853 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.698404 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.698417 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.699749 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.701617 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.701620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.701621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.701806 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.702119 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.702122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.702124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.702219 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.702565 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.702568 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.702655 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.702738 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.702868 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260929 18:38:17.703042 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.694805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.0.198.61:33129)
0929 18:38:17.694930 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.694934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.694947 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:17.694986 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.694989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.695028 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.695083 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.695486 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.696263 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.697320 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.697651 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.697758 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.699904 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.699908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.699910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.699938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.701454 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.701908 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.701911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.701913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.702021 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.702293 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.702296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.702298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.702477 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.702560 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.702798 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.702898 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:38:17.703758 14344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:38:17.703964 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.694546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.0.198.62:39355)
0929 18:38:17.694694 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.694698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.694710 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:17.694800 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.694803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.694841 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.694905 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.695541 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.696255 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.697320 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.697685 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.697763 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.700397 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.700400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.700403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.700429 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.701923 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.702506 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.702510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.702512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.702587 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.702844 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.702847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.702849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.703036 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.703483 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.703570 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.703677 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
I20260929 18:38:17.709862 14269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } attempt: 3
I20260929 18:38:17.712013 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.704215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57256)
0929 18:38:17.704302 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.704316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.704382 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.704592 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.704595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.704732 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.704793 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.704797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.705615 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.705618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.706177 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.706186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.707036 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.707039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.707148 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.707561 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.707575 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.708765 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.710599 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.710602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.710603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.710794 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.711051 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.711054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.711055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.711126 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.711540 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.711543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.711620 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.711708 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.711841 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":18}
I20260929 18:38:17.712014 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.704348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.0.198.60:44185)
0929 18:38:17.704429 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.704433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.704443 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:38:17.704468 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.704471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.704513 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.704570 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.704882 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.705527 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.706272 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.706936 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.707026 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.708869 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.708873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.708876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.708896 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.710489 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.710890 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.710893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.710895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.710967 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.711208 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.711211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.711212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.711422 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.711510 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.711766 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.711898 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":31}
I20260929 18:38:17.713186 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.704973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.0.198.61:33129)
0929 18:38:17.705084 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.705088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.705100 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:17.705131 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.705134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.705171 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.705243 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.705777 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.706318 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.707024 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.707227 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.707308 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.709734 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.709738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.709741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.709766 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.711534 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.711976 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.711980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.711981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.712061 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.712331 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.712334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.712335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.712592 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.712801 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.712938 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.713041 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":45}
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:38:17.715764 14344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca: Bootstrap starting.
I20260929 18:38:17.718714 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.710300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57272)
0929 18:38:17.710408 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.710422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.710492 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.711132 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.711136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.711309 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.711384 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.711389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.712353 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.712355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.713101 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.713110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.713767 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.713771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.713835 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.714308 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.714321 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.715573 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.717213 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.717216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.717218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.717452 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.717739 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.717742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.717744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.717832 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.718218 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.718221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.718321 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.718430 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.718562 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:38:17.718699 14344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:17.719291 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.710535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.0.198.60:44185)
0929 18:38:17.710935 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.710941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.710952 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:38:17.710979 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.710981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.711023 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.711095 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.711481 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.712249 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.713338 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.713652 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.714226 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.715721 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.715724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.715726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.715746 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.717111 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.717568 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.717571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.717572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.717648 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.717909 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.717912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.717913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.718133 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.718207 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.719012 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.719136 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":113,"threads_started":1}
I20260929 18:38:17.719614 14344 log.cc:824] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:17.720113 14194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } attempt: 3
I20260929 18:38:17.720419 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:38:17.722484 14269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: Bootstrap starting.
I20260929 18:38:17.724265 14344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca: No bootstrap required, opened a new log
I20260929 18:38:17.725294 14269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:17.726192 14269 log.cc:824] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:17.728319 14269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: No bootstrap required, opened a new log
I20260929 18:38:17.728315 14344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.728556 14344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:17.728606 14344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e0708f1e74426982cda6f2eb89e8ca, State: Initialized, Role: FOLLOWER
I20260929 18:38:17.729257 14344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.730747 14344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:17.732249 14269 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.732463 14269 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:17.732504 14269 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc1d964af2e49d3a3825666f7fd083f, State: Initialized, Role: FOLLOWER
I20260929 18:38:17.732726 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.723224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.0.198.62:39355)
0929 18:38:17.723356 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.723360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.723978 (+ 618us) server_negotiation.cc:407] Connection header received
0929 18:38:17.724147 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.724150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.724194 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.724241 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.725179 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.725923 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.727068 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.727416 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.727488 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.729211 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.729214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.729217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.729245 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.730856 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.731323 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.731326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.731328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.731433 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.731751 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.731755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.731757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.731977 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.732075 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.732444 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.732569 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:38:17.734076 14269 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.734583 14371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.734826 14371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.735535 14269 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 18:38:17 dist-test-slave-q5qx krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:38:17.735837 14275 master_runner.cc:186] Error getting master registration for 127.0.198.62:39355: OK, instance_id { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" instance_seqno: 1790707097258131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:38:17.739884 14275 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:38:17.741129 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.720494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:44185 (local address 127.0.0.1:57286)
0929 18:38:17.720594 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.720609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.720684 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.722474 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.722477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.722665 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.722760 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.722766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.733640 (+ 10874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.733644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.734426 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.734437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.735418 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.735421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.735549 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.735909 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:17.735923 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.737411 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.739438 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.739441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.739444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.739677 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.740041 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:17.740044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:17.740046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:17.740139 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:17.740644 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.740647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.740744 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.740826 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.740969 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:38:17.741834 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.722118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.198.60:44185)
0929 18:38:17.722219 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.722223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.722237 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:38:17.722272 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.722275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.722315 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.732795 (+ 10480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.732900 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.733533 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.734865 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.735303 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.735402 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.737560 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.737563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.737566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.737593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.739318 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.739826 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.739829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.739831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.739911 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.740296 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.740299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.740301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.740504 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.741443 (+ 939us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.741518 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.741641 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 18:38:17.741835 14382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:38:17.742162 14382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:38:17.742403 14359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.742530 14359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.739012 14201 master_runner.cc:186] Error getting master registration for 127.0.198.62:39355: OK, instance_id { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" instance_seqno: 1790707097258131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:38:17.742674 14201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:38:17.745061 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: Bootstrap starting.
W20260929 18:38:17.745792 14370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:38:17.745843 14370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:38:17.745954 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.723496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.0.198.61:33129)
0929 18:38:17.723596 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.723600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.724185 (+ 585us) server_negotiation.cc:407] Connection header received
0929 18:38:17.724619 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.724623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.724664 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.724708 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.728004 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.728670 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.730813 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.731144 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.731218 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.733880 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.733884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.733887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.733912 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.736534 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.743600 (+ 7066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.743604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.743606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.743693 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.744370 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.744373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.744375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.744591 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.744717 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.745691 (+ 974us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.745814 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:38:17.747807 14194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:17.748138 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.723045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.198.60:44185)
0929 18:38:17.723142 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.723146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.724004 (+ 858us) server_negotiation.cc:407] Connection header received
0929 18:38:17.724402 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.724405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.724445 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.724488 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.726305 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.726977 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.729777 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.730128 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.730201 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.736735 (+ 6534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.736741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.736744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:17.736772 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.738477 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.743992 (+ 5515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.743995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.743997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.744076 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:17.745036 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:17.745039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:17.745041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:17.745241 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.745342 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.747868 (+ 2526us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.747994 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:38:17.748787 14194 log.cc:824] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:17.751021 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: No bootstrap required, opened a new log
I20260929 18:38:17.755512 14194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.755741 14194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:17.755795 14194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26028a3fce954df3a0c0a06cad15ffc4, State: Initialized, Role: FOLLOWER
I20260929 18:38:17.756330 14194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.756863 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.757094 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.757757 14194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:38:17.764760 14397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:38:17.764855 14397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:38:17.766552 14126 master_runner.cc:263] No leader master found from master 26028a3fce954df3a0c0a06cad15ffc4
I20260929 18:38:17.766628 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:38:17.850168 14359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:17.850890 14359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.852075 14359 leader_election.cc:288] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26028a3fce954df3a0c0a06cad15ffc4 (127.0.198.62:39355), ddc1d964af2e49d3a3825666f7fd083f (127.0.198.61:33129)
I20260929 18:38:17.852922 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40e0708f1e74426982cda6f2eb89e8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc1d964af2e49d3a3825666f7fd083f" is_pre_election: true
I20260929 18:38:17.853247 14243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e0708f1e74426982cda6f2eb89e8ca in term 0.
I20260929 18:38:17.854049 14295 leader_election.cc:302] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e0708f1e74426982cda6f2eb89e8ca, ddc1d964af2e49d3a3825666f7fd083f; no voters:
I20260929 18:38:17.854444 14359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:38:17.854529 14359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:38:17.854573 14359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:17.855914 14359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.856770 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40e0708f1e74426982cda6f2eb89e8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26028a3fce954df3a0c0a06cad15ffc4" is_pre_election: true
I20260929 18:38:17.857110 14168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e0708f1e74426982cda6f2eb89e8ca in term 0.
I20260929 18:38:17.857893 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40e0708f1e74426982cda6f2eb89e8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26028a3fce954df3a0c0a06cad15ffc4"
I20260929 18:38:17.858070 14168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:17.859392 14168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e0708f1e74426982cda6f2eb89e8ca in term 1.
I20260929 18:38:17.859798 14292 leader_election.cc:302] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26028a3fce954df3a0c0a06cad15ffc4, 40e0708f1e74426982cda6f2eb89e8ca; no voters:
I20260929 18:38:17.860222 14359 leader_election.cc:288] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [CANDIDATE]: Term 1 election: Requested vote from peers 26028a3fce954df3a0c0a06cad15ffc4 (127.0.198.62:39355), ddc1d964af2e49d3a3825666f7fd083f (127.0.198.61:33129)
I20260929 18:38:17.860693 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40e0708f1e74426982cda6f2eb89e8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc1d964af2e49d3a3825666f7fd083f"
I20260929 18:38:17.860831 14243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:17.862146 14243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e0708f1e74426982cda6f2eb89e8ca in term 1.
I20260929 18:38:17.862649 14359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:17.862814 14359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [term 1 LEADER]: Becoming Leader. State: Replica: 40e0708f1e74426982cda6f2eb89e8ca, State: Running, Role: LEADER
I20260929 18:38:17.863232 14359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } }
I20260929 18:38:17.865526 14359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e0708f1e74426982cda6f2eb89e8ca. Latest consensus state: current_term: 1 leader_uuid: "40e0708f1e74426982cda6f2eb89e8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.865700 14359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:17.866197 14401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:17.867276 14401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:17.874692 14243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [term 1 FOLLOWER]: Refusing update from remote peer 40e0708f1e74426982cda6f2eb89e8ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:17.875542 14399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:17.881435 14371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e0708f1e74426982cda6f2eb89e8ca. Latest consensus state: current_term: 1 leader_uuid: "40e0708f1e74426982cda6f2eb89e8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.881561 14371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.883656 14168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [term 1 FOLLOWER]: Refusing update from remote peer 40e0708f1e74426982cda6f2eb89e8ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:17.884272 14399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:17.887861 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e0708f1e74426982cda6f2eb89e8ca. Latest consensus state: current_term: 1 leader_uuid: "40e0708f1e74426982cda6f2eb89e8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.887987 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.895309 14399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40e0708f1e74426982cda6f2eb89e8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.895460 14399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:17.896926 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40e0708f1e74426982cda6f2eb89e8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.897082 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.900137 14402 mvcc.cc:205] Tried to move back new op lower bound from 7334736272885735424 to 7334736272852770816. Current Snapshot: MvccSnapshot[applied={T|T < 7334736272885735424}]
I20260929 18:38:17.901175 14359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40e0708f1e74426982cda6f2eb89e8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.901289 14359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:17.902807 14371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40e0708f1e74426982cda6f2eb89e8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26028a3fce954df3a0c0a06cad15ffc4" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39355 } } peers { permanent_uuid: "ddc1d964af2e49d3a3825666f7fd083f" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 33129 } } peers { permanent_uuid: "40e0708f1e74426982cda6f2eb89e8ca" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 44185 } } }
I20260929 18:38:17.902936 14371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.903911 14405 mvcc.cc:205] Tried to move back new op lower bound from 7334736272885735424 to 7334736272852770816. Current Snapshot: MvccSnapshot[applied={T|T < 7334736272885735424}]
I20260929 18:38:17.904146 14403 mvcc.cc:205] Tried to move back new op lower bound from 7334736272885735424 to 7334736272852770816. Current Snapshot: MvccSnapshot[applied={T|T < 7334736272885735424}]
I20260929 18:38:17.909511 14401 catalog_manager.cc:1347] Generated new cluster ID: e2fa4812432249609a730fbab6bb2548
I20260929 18:38:17.909586 14401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:17.931457 14401 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:17.932449 14401 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:17.944916 14401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40e0708f1e74426982cda6f2eb89e8ca: Generated new TSK 0
I20260929 18:38:17.945904 14401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:38:18.743597 14382 catalog_manager.cc:1259] Loaded cluster ID: e2fa4812432249609a730fbab6bb2548
I20260929 18:38:18.743685 14382 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: loading cluster ID for follower catalog manager: success
I20260929 18:38:18.745436 14382 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: acquiring CA information for follower catalog manager: success
I20260929 18:38:18.746753 14382 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddc1d964af2e49d3a3825666f7fd083f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:38:18.766191 14397 catalog_manager.cc:1259] Loaded cluster ID: e2fa4812432249609a730fbab6bb2548
I20260929 18:38:18.766279 14397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: loading cluster ID for follower catalog manager: success
I20260929 18:38:18.767921 14397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: acquiring CA information for follower catalog manager: success
I20260929 18:38:18.769131 14397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 26028a3fce954df3a0c0a06cad15ffc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:38:19.011859 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14126
I20260929 18:38:19.017295 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14201
I20260929 18:38:19.022671 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14275
2026-09-29T18:38:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3039 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:38:19 dist-test-slave-q5qx krb5kdc[14416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:38:19 dist-test-slave-q5qx krb5kdc[14416](info): set up 2 sockets
Sep 29 18:38:19 dist-test-slave-q5qx krb5kdc[14416](info): commencing operation
krb5kdc: starting...
W20260929 18:38:19.986135 792 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:38:20 dist-test-slave-q5qx krb5kdc[14416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:38:20.106238 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:43519
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:20.213707 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:20.214023 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:20.214079 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:20.218146 14429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:20.218209 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:20.218227 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:20.218243 14429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:20.218257 14429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:20.223600 14429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:20.224678 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:20.225647 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:20.234064 14437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:20.234123 14429 server_base.cc:1034] running on GCE node
W20260929 18:38:20.234063 14435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:20.234084 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:20.234956 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:20.236089 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:20.237264 14429 hybrid_clock.cc:648] HybridClock initialized: now 1790707100237248 us; error 32 us; skew 500 ppm
Sep 29 18:38:20 dist-test-slave-q5qx krb5kdc[14416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:20.240146 14429 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:20.241073 14429 webserver.cc:460] Webserver started at http://127.0.198.62:42005/ using document root <none> and password file <none>
I20260929 18:38:20.241605 14429 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:20.241667 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:20.241851 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:20.244027 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "05a5411cb9ec437bb35001028a5ad1e2"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-q5qx"
server_key: "d5777c5cf115e377b3a2b1141ce80d43"
server_key_iv: "5a1eee878ce36bb79920c8465850bb50"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.244544 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "05a5411cb9ec437bb35001028a5ad1e2"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-q5qx"
server_key: "d5777c5cf115e377b3a2b1141ce80d43"
server_key_iv: "5a1eee878ce36bb79920c8465850bb50"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.248427 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:38:20.250944 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.252209 14429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:20.252315 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "05a5411cb9ec437bb35001028a5ad1e2"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-q5qx"
server_key: "d5777c5cf115e377b3a2b1141ce80d43"
server_key_iv: "5a1eee878ce36bb79920c8465850bb50"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.252408 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:20.265308 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:20.268319 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:20.268476 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:20.275260 14429 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:43519
I20260929 18:38:20.275286 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:43519 every 8 connection(s)
I20260929 18:38:20.276199 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:20.279007 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:20.282407 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14429
I20260929 18:38:20.282533 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:38:20.282819 792 external_mini_cluster.cc:1428] Setting key ff5d5676db3fc95d99889b3e36c22769
I20260929 18:38:20.284946 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2: Bootstrap starting.
I20260929 18:38:20.287164 14497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:20.287894 14497 log.cc:824] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:20.289705 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2: No bootstrap required, opened a new log
Sep 29 18:38:20 dist-test-slave-q5qx krb5kdc[14416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:20.293349 14497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a5411cb9ec437bb35001028a5ad1e2" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43519 } }
I20260929 18:38:20.293588 14497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:20.293663 14497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a5411cb9ec437bb35001028a5ad1e2, State: Initialized, Role: FOLLOWER
I20260929 18:38:20.294134 14497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a5411cb9ec437bb35001028a5ad1e2" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43519 } }
I20260929 18:38:20.294275 14497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:20.294341 14497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:20.294438 14497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:20.295396 14497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a5411cb9ec437bb35001028a5ad1e2" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43519 } }
I20260929 18:38:20.295728 14497 leader_election.cc:302] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05a5411cb9ec437bb35001028a5ad1e2; no voters:
I20260929 18:38:20.296032 14497 leader_election.cc:288] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:20.296082 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.284213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.198.62:43519)
0929 18:38:20.284658 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.284673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.284709 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:38:20.285373 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.285392 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.285735 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.286062 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:20.286834 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.287865 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.288710 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.289166 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.289302 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.291623 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:20.291628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:20.291641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:20.291667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:20.293252 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.293647 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.293652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.293656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.293776 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.294079 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.294082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.294083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.294521 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:20.294703 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.294987 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.295275 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260929 18:38:20.296150 14502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:20.297323 14502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 05a5411cb9ec437bb35001028a5ad1e2, State: Running, Role: LEADER
I20260929 18:38:20.297320 14497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:20.297693 14502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a5411cb9ec437bb35001028a5ad1e2" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43519 } }
I20260929 18:38:20.299894 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:38:20.299926 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05a5411cb9ec437bb35001028a5ad1e2. Latest consensus state: current_term: 1 leader_uuid: "05a5411cb9ec437bb35001028a5ad1e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a5411cb9ec437bb35001028a5ad1e2" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43519 } } }
I20260929 18:38:20.300029 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:20.299590 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05a5411cb9ec437bb35001028a5ad1e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a5411cb9ec437bb35001028a5ad1e2" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43519 } } }
I20260929 18:38:20.300256 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2 [sys.catalog]: This master's current role is: LEADER
W20260929 18:38:20.302330 14516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:38:20.302397 14516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:38:20.302482 14517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:20.303391 14517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:20.308702 14517 catalog_manager.cc:1347] Generated new cluster ID: d4f050d24e364d06987fdbb37c6cf64c
I20260929 18:38:20.308770 14517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:20.327244 14517 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:20.328332 14517 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:20.337666 14517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05a5411cb9ec437bb35001028a5ad1e2: Generated new TSK 0
I20260929 18:38:20.338435 14517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:38:20.356335 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14429
2026-09-29T18:38:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1332 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:38:20 dist-test-slave-q5qx krb5kdc[14524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:38:20 dist-test-slave-q5qx krb5kdc[14524](info): set up 2 sockets
Sep 29 18:38:20 dist-test-slave-q5qx krb5kdc[14524](info): commencing operation
krb5kdc: starting...
W20260929 18:38:21.314064 792 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:38:21 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:38:21.437947 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41069
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:21.547945 14537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:21.548247 14537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:21.548287 14537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:21.552387 14537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:21.552464 14537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:21.552481 14537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:21.552497 14537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:21.552511 14537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:21.558262 14537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:21.559348 14537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:21.560343 14537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:21.569227 14542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:21.569219 14545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:21.569223 14543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:21.569334 14537 server_base.cc:1034] running on GCE node
I20260929 18:38:21.570176 14537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:21.571359 14537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:21.572592 14537 hybrid_clock.cc:648] HybridClock initialized: now 1790707101572573 us; error 33 us; skew 500 ppm
Sep 29 18:38:21 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:21.580986 14537 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:21.582121 14537 webserver.cc:460] Webserver started at http://127.0.198.62:43071/ using document root <none> and password file <none>
I20260929 18:38:21.582712 14537 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:21.582782 14537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:21.582988 14537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:21.585222 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "62fe08d8657f4356b1de6e338d4cd3a3"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "bc35921113f5c93afe0f7c3e77ec3cf5"
server_key_iv: "58fab3a55c53e52dfb40946abebd76d8"
server_key_version: "encryptionkey@0"
I20260929 18:38:21.585770 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "62fe08d8657f4356b1de6e338d4cd3a3"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "bc35921113f5c93afe0f7c3e77ec3cf5"
server_key_iv: "58fab3a55c53e52dfb40946abebd76d8"
server_key_version: "encryptionkey@0"
I20260929 18:38:21.589790 14537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:38:21.592559 14552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:21.594027 14537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:21.594166 14537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "62fe08d8657f4356b1de6e338d4cd3a3"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "bc35921113f5c93afe0f7c3e77ec3cf5"
server_key_iv: "58fab3a55c53e52dfb40946abebd76d8"
server_key_version: "encryptionkey@0"
I20260929 18:38:21.594282 14537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:21.627282 14537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:21.630548 14537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:21.630733 14537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:21.637643 14537 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:41069
I20260929 18:38:21.637953 14604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:41069 every 8 connection(s)
I20260929 18:38:21.638636 14537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:21.641422 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:21.644871 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14537
I20260929 18:38:21.644994 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:38:21.645244 792 external_mini_cluster.cc:1428] Setting key 961fb83b39dfe310d42556145dc616df
I20260929 18:38:21.647243 14605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Bootstrap starting.
I20260929 18:38:21.649691 14605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:21.650439 14605 log.cc:824] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:21.652447 14605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: No bootstrap required, opened a new log
Sep 29 18:38:21 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:21.656288 14605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:21.656533 14605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:21.656584 14605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fe08d8657f4356b1de6e338d4cd3a3, State: Initialized, Role: FOLLOWER
I20260929 18:38:21.657002 14605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:21.657123 14605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:21.657183 14605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:21.657274 14605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:21.658195 14605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:21.658527 14605 leader_election.cc:302] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62fe08d8657f4356b1de6e338d4cd3a3; no voters:
I20260929 18:38:21.658880 14605 leader_election.cc:288] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:21.658967 14610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:21.659173 14610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 LEADER]: Becoming Leader. State: Replica: 62fe08d8657f4356b1de6e338d4cd3a3, State: Running, Role: LEADER
I20260929 18:38:21.659539 14610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:21.659679 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:21.646617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.0.198.62:41069)
0929 18:38:21.647020 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:21.647036 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:21.647071 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:38:21.647727 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:21.647746 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:21.648100 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:21.648386 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:21.649250 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:21.650363 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:21.651342 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:21.651875 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:21.652002 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:21.654734 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:21.654739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:21.654753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:21.654792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:21.656657 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:21.657080 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:21.657086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:21.657092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:21.657239 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:21.657517 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:21.657520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:21.657522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:21.657935 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:21.658150 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:21.658458 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 18:38:21.658757 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
I20260929 18:38:21.660223 14605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:21.661175 14612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } } }
I20260929 18:38:21.661298 14612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:21.661653 14611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fe08d8657f4356b1de6e338d4cd3a3. Latest consensus state: current_term: 1 leader_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } } }
I20260929 18:38:21.662305 14611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: This master's current role is: LEADER
W20260929 18:38:21.664865 14626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:38:21.664928 14626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:38:21.664999 14619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:21.665660 14619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:21.671301 14619 catalog_manager.cc:1347] Generated new cluster ID: 4b827cb07bc044db9111dfa34d3078e6
I20260929 18:38:21.671378 14619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:21.704257 14619 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:21.705228 14619 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:21.713394 14619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Generated new TSK 0
I20260929 18:38:21.714241 14619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:38:21.757846 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41069
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:21.869995 14631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:21.870294 14631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:21.870342 14631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:21.874675 14631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:21.874755 14631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:21.874825 14631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:38:21.880285 14631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:21.881349 14631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:21.882457 14631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:21.892269 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:21.892277 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:21.892287 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:21.892340 14631 server_base.cc:1034] running on GCE node
I20260929 18:38:21.892958 14631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:21.893594 14631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:21.894773 14631 hybrid_clock.cc:648] HybridClock initialized: now 1790707101894758 us; error 30 us; skew 500 ppm
Sep 29 18:38:21 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:21.897629 14631 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:38:21.898622 14631 webserver.cc:460] Webserver started at http://127.0.198.1:37119/ using document root <none> and password file <none>
I20260929 18:38:21.899187 14631 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:21.899248 14631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:21.899421 14631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:21.901453 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "456b28faff91441598be5a2d79886ef6"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "f612f50ebe1ebbf0c6364a484a39173c"
server_key_iv: "3c2a251abee3b65f0e704b68fa99c075"
server_key_version: "encryptionkey@0"
I20260929 18:38:21.902006 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "456b28faff91441598be5a2d79886ef6"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "f612f50ebe1ebbf0c6364a484a39173c"
server_key_iv: "3c2a251abee3b65f0e704b68fa99c075"
server_key_version: "encryptionkey@0"
I20260929 18:38:21.906014 14631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:38:21.908646 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:21.910077 14631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:21.910188 14631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "456b28faff91441598be5a2d79886ef6"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "f612f50ebe1ebbf0c6364a484a39173c"
server_key_iv: "3c2a251abee3b65f0e704b68fa99c075"
server_key_version: "encryptionkey@0"
I20260929 18:38:21.910287 14631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:21.924209 14631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:21.927556 14631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:21.927723 14631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:21.928299 14631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:21.929107 14631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:21.929155 14631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:21.929194 14631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:21.929208 14631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:21.937997 14631 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:41225
I20260929 18:38:21.938030 14759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:41225 every 8 connection(s)
I20260929 18:38:21.938927 14631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:21.944360 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14631
I20260929 18:38:21.944506 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:38:21.944800 792 external_mini_cluster.cc:1428] Setting key dc38df24943491daec1c606260133d16
Sep 29 18:38:21 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:38:21.953743 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:21.940774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:59489 (local address 127.0.198.62:41069)
0929 18:38:21.940942 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:21.940946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:21.941782 (+ 836us) server_negotiation.cc:407] Connection header received
0929 18:38:21.943325 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:21.943337 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:21.943390 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:21.943472 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:21.945119 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:21.945760 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:21.947150 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:21.947391 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:21.947482 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:21.950173 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:21.950176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:21.950178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:21.950205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:21.951569 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:21.952122 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:21.952125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:21.952126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:21.952206 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:21.952547 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:21.952550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:21.952551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:21.952745 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:21.952832 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:21.953381 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 18:38:21.953574 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:38:21.954588 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:21.941165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:59489)
0929 18:38:21.941624 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:21.941668 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:21.943049 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:21.943703 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:21.943711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:21.944224 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:21.944942 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:21.944963 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:21.945891 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:21.945895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:21.947024 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:21.947034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:21.947543 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:21.947546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:21.947718 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:21.948336 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:21.948353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:21.949943 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:21.951696 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:21.951703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:21.951722 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:21.951990 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:21.952291 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:21.952295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:21.952297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:21.952444 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:21.952847 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:21.952854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:21.953137 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 18:38:21.953306 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:38:21.953672 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:38:21.955856 14760 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41069
I20260929 18:38:21.956100 14760 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:21.956595 14760 heartbeater.cc:511] Master 127.0.198.62:41069 requested a full tablet report, sending...
I20260929 18:38:21.958105 14569 ts_manager.cc:194] Registered new tserver with Master: 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225)
I20260929 18:38:21.959230 14569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:59489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:38:21.973760 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41069
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:22.081606 14765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:22.081877 14765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:22.081920 14765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:22.086002 14765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:22.086074 14765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:22.086155 14765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:38:22.091596 14765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:22.092633 14765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:22.093618 14765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:22.102905 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:22.103001 14770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:22.103018 14771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:22.103298 14765 server_base.cc:1034] running on GCE node
I20260929 18:38:22.103745 14765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:22.104349 14765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:22.105556 14765 hybrid_clock.cc:648] HybridClock initialized: now 1790707102105519 us; error 41 us; skew 500 ppm
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:22.108335 14765 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:38:22.109282 14765 webserver.cc:460] Webserver started at http://127.0.198.2:40989/ using document root <none> and password file <none>
I20260929 18:38:22.109820 14765 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:22.109884 14765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:22.110163 14765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:22.113039 14765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "c7a40c8f558a468e87fe0e6438b8d424"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "0348692a8ac11d0515af282f26449ad5"
server_key_iv: "9edcd1b9930eeae594939185cd2bc1dc"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.113715 14765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "c7a40c8f558a468e87fe0e6438b8d424"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "0348692a8ac11d0515af282f26449ad5"
server_key_iv: "9edcd1b9930eeae594939185cd2bc1dc"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.117671 14765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 18:38:22.120189 14780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.121547 14765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:38:22.121687 14765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "c7a40c8f558a468e87fe0e6438b8d424"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "0348692a8ac11d0515af282f26449ad5"
server_key_iv: "9edcd1b9930eeae594939185cd2bc1dc"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.121798 14765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:22.146847 14765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:22.150214 14765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:22.150403 14765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:22.150988 14765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:22.151809 14765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:22.151857 14765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.151909 14765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:22.151937 14765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.160498 14765 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:35537
I20260929 18:38:22.160537 14893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:35537 every 8 connection(s)
I20260929 18:38:22.161535 14765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:22.170329 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14765
I20260929 18:38:22.170428 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:38:22.170711 792 external_mini_cluster.cc:1428] Setting key 29624300a0eb372f3f8502050c6eb0ff
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:22.175083 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.163358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:40031 (local address 127.0.198.62:41069)
0929 18:38:22.163515 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.163519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.164312 (+ 793us) server_negotiation.cc:407] Connection header received
0929 18:38:22.165593 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.165596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.165655 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.165730 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:22.166979 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.167508 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.168632 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.168856 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.168953 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.171535 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:22.171538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:22.171540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:22.171566 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:22.172953 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.173478 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.173481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.173482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.173565 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.173909 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.173910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.173912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.174142 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:22.174216 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.174839 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.174946 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:38:22.176275 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.163731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:40031)
0929 18:38:22.164161 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.164205 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.165375 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.165927 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.165934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.166331 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:22.166830 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.166843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.167630 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.167633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.168514 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.168524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.168997 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.169001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.169162 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.169712 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:22.169728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:22.171347 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:22.173059 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:22.173065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:22.173082 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:22.173366 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:22.173646 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:22.173649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:22.173651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:22.173833 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:22.174225 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:22.174232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:22.174581 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.174787 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.175118 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260929 18:38:22.177526 14894 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41069
I20260929 18:38:22.177793 14894 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:22.178400 14894 heartbeater.cc:511] Master 127.0.198.62:41069 requested a full tablet report, sending...
I20260929 18:38:22.179605 14569 ts_manager.cc:194] Registered new tserver with Master: c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537)
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:38:22.180356 14569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:40031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:38:22.197328 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41069
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:22.305815 14899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:22.306107 14899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:22.306152 14899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:22.310133 14899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:22.310221 14899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:22.310287 14899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:38:22.315752 14899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:22.316756 14899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:22.317728 14899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:22.327180 14907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:22.327186 14904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:22.327247 14899 server_base.cc:1034] running on GCE node
W20260929 18:38:22.327196 14905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:22.327821 14899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:22.328401 14899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:22.329612 14899 hybrid_clock.cc:648] HybridClock initialized: now 1790707102329593 us; error 30 us; skew 500 ppm
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:22.332355 14899 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:38:22.333232 14899 webserver.cc:460] Webserver started at http://127.0.198.3:41863/ using document root <none> and password file <none>
I20260929 18:38:22.333755 14899 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:22.333820 14899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:22.334064 14899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:22.336097 14899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "c3a43de10c0b45aebf1284e1f1b41a40"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "c20f0c281847c500ff9066f1cab0bc7b"
server_key_iv: "c4db0e6e25fb98dea892815b6c24c77c"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.336627 14899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "c3a43de10c0b45aebf1284e1f1b41a40"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "c20f0c281847c500ff9066f1cab0bc7b"
server_key_iv: "c4db0e6e25fb98dea892815b6c24c77c"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.340579 14899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:38:22.343170 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.344455 14899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:22.344586 14899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "c3a43de10c0b45aebf1284e1f1b41a40"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "c20f0c281847c500ff9066f1cab0bc7b"
server_key_iv: "c4db0e6e25fb98dea892815b6c24c77c"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.344687 14899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:22.383491 14899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:22.386767 14899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:22.386942 14899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:22.387573 14899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:22.388391 14899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:22.388437 14899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.388499 14899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:22.388530 14899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.396896 14899 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:39505
I20260929 18:38:22.396920 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:39505 every 8 connection(s)
I20260929 18:38:22.397799 14899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:22.404235 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 14899
I20260929 18:38:22.404356 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:38:22.404662 792 external_mini_cluster.cc:1428] Setting key e8252602326def2ad5ba4cdbe09a9651
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:22.412973 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.399717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:46921 (local address 127.0.198.62:41069)
0929 18:38:22.399891 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.399896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.400971 (+ 1075us) server_negotiation.cc:407] Connection header received
0929 18:38:22.402325 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.402327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.402375 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.402456 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:22.403703 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.404408 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.405726 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.406028 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.406131 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.408848 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:22.408852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:22.408854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:22.408883 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:22.410495 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.411169 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.411172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.411174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.411259 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.411792 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.411795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.411796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.411996 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:22.412084 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.412697 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.412838 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:38:22.413945 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.400281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:46921)
0929 18:38:22.400805 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.400849 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.402105 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.402661 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.402668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.403047 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:22.403560 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.403577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.404520 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.404524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.405624 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.405634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.406197 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.406201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.406401 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.407117 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:22.407134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:22.408662 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:22.410671 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:22.410677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:22.410692 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:22.410982 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:22.411364 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:22.411366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:22.411368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:22.411619 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:22.412113 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:22.412119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:22.412412 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.412634 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.412925 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":113,"threads_started":1}
I20260929 18:38:22.415225 15028 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41069
I20260929 18:38:22.415460 15028 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:22.415949 15028 heartbeater.cc:511] Master 127.0.198.62:41069 requested a full tablet report, sending...
I20260929 18:38:22.417083 14569 ts_manager.cc:194] Registered new tserver with Master: c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505)
I20260929 18:38:22.417804 14569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:46921
I20260929 18:38:22.418887 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:38:22.428117 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.420759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.198.62:41069)
0929 18:38:22.420889 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.420894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.421011 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:38:22.421116 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.421118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.421155 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.421215 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:22.421923 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.422477 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.423435 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.423659 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.423738 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.425092 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:22.425095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:22.425097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:22.425120 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:22.426495 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.426886 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.426889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.426890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.426954 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.427200 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.427203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.427204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.427429 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:22.427533 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.427854 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.427959 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:38:22.431015 14569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:38:22.434194 14569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:38:22.448076 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.442739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39505 (local address 127.0.0.1:47244)
0929 18:38:22.443120 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.443168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.443293 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.443977 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.443982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.444011 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:22.444115 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.444121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.445286 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.445290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.446702 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.446712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.447583 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.447586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.447689 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.447726 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.447837 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.447950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 18:38:22.448663 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.442855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.0.198.3:39505)
0929 18:38:22.443358 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.443377 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.443400 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:38:22.443467 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.443471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.443683 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.443831 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:22.444219 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.445147 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.446836 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.447474 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.447580 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.447727 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.448411 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.448514 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":177,"threads_started":1}
I20260929 18:38:22.448928 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.443474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35537 (local address 127.0.0.1:44784)
0929 18:38:22.443689 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.443701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.443782 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.444908 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.444910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.444928 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:22.445018 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.445024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.446139 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.446142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.447151 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.447161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.448562 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.448566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.448657 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.448680 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.448729 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.448794 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260929 18:38:22.449331 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.443796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41225 (local address 127.0.0.1:36700)
0929 18:38:22.444004 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.444017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.444096 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.445242 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.445244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.445262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:22.445375 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.445381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.446777 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.446780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.447785 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.447793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.448978 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.448982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.449064 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.449085 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.449147 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.449212 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":22,"thread_start_us":91,"threads_started":1}
I20260929 18:38:22.449610 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.444205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.198.2:35537)
0929 18:38:22.444487 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.444499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.444520 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:38:22.444596 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.444601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.444785 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.444922 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:22.445105 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.446015 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.447402 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.448392 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.449188 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.449318 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.449368 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.449474 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260929 18:38:22.449811 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.444554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.0.198.1:41225)
0929 18:38:22.444837 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.444851 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.444871 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:38:22.444948 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.444953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.445114 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.445262 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:22.445485 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.446644 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.447875 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.448676 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.448770 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.448907 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.449585 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.449683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
I20260929 18:38:22.451643 14962 tablet_service.cc:1467] Processing CreateTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c (DEFAULT_TABLE table=test-table [id=243d916bb25f49fe92b1c09b8589889f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:22.452040 14828 tablet_service.cc:1467] Processing CreateTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c (DEFAULT_TABLE table=test-table [id=243d916bb25f49fe92b1c09b8589889f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:22.452625 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7ab6960c884dbdb98d871ec9ce202c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:22.452907 14828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7ab6960c884dbdb98d871ec9ce202c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:22.456096 14694 tablet_service.cc:1467] Processing CreateTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c (DEFAULT_TABLE table=test-table [id=243d916bb25f49fe92b1c09b8589889f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:22.457149 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7ab6960c884dbdb98d871ec9ce202c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:22.458737 15045 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Bootstrap starting.
I20260929 18:38:22.460275 15046 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Bootstrap starting.
I20260929 18:38:22.461751 15045 tablet_bootstrap.cc:654] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:22.462675 15045 log.cc:824] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:22.462971 15046 tablet_bootstrap.cc:654] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:22.463797 15046 log.cc:824] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:22.464840 15045 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: No bootstrap required, opened a new log
I20260929 18:38:22.464937 15048 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Bootstrap starting.
I20260929 18:38:22.465047 15045 ts_tablet_manager.cc:1397] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:38:22.465788 15046 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: No bootstrap required, opened a new log
I20260929 18:38:22.466022 15046 ts_tablet_manager.cc:1397] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:38:22.467195 15048 tablet_bootstrap.cc:654] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:22.468014 15048 log.cc:824] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:22.468127 15045 raft_consensus.cc:348] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.468385 15045 raft_consensus.cc:374] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:22.468443 15045 raft_consensus.cc:729] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a43de10c0b45aebf1284e1f1b41a40, State: Initialized, Role: FOLLOWER
I20260929 18:38:22.468909 15045 consensus_queue.cc:260] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.469590 15028 heartbeater.cc:503] Master 127.0.198.62:41069 was elected leader, sending a full tablet report...
I20260929 18:38:22.469681 15046 raft_consensus.cc:348] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.469937 15046 raft_consensus.cc:374] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:22.469950 15045 ts_tablet_manager.cc:1428] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:38:22.470017 15046 raft_consensus.cc:729] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a40c8f558a468e87fe0e6438b8d424, State: Initialized, Role: FOLLOWER
I20260929 18:38:22.470566 15048 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: No bootstrap required, opened a new log
I20260929 18:38:22.470485 15046 consensus_queue.cc:260] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.470825 15048 ts_tablet_manager.cc:1397] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260929 18:38:22.471437 15046 ts_tablet_manager.cc:1428] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:38:22.471815 14894 heartbeater.cc:503] Master 127.0.198.62:41069 was elected leader, sending a full tablet report...
I20260929 18:38:22.474789 15048 raft_consensus.cc:348] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.475018 15048 raft_consensus.cc:374] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:22.475076 15048 raft_consensus.cc:729] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456b28faff91441598be5a2d79886ef6, State: Initialized, Role: FOLLOWER
I20260929 18:38:22.475425 15048 consensus_queue.cc:260] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.476023 14760 heartbeater.cc:503] Master 127.0.198.62:41069 was elected leader, sending a full tablet report...
I20260929 18:38:22.476276 15048 ts_tablet_manager.cc:1428] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260929 18:38:22.487706 15052 raft_consensus.cc:482] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:22.487903 15052 raft_consensus.cc:504] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.488955 15052 leader_election.cc:288] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505), 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225)
I20260929 18:38:22.493535 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.489300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:60443 (local address 127.0.198.3:39505)
0929 18:38:22.489422 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.489426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.489442 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:38:22.489594 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.489599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.489651 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.489719 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:22.490117 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.490854 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.492359 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.492964 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.493064 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.493127 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.493340 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.493407 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:38:22.493577 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.489646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:37955 (local address 127.0.198.1:41225)
0929 18:38:22.489751 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.489755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.489933 (+ 178us) server_negotiation.cc:407] Connection header received
0929 18:38:22.490070 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.490073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.490122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.490184 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:22.490438 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.491066 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.492366 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.492964 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.493064 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.493127 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.493344 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.493407 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:38:22.493578 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.489486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41225 (local address 127.0.198.2:37955)
0929 18:38:22.489841 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.489851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.489938 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.490223 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.490225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.490242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:22.490347 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.490352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.491156 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.491159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.492269 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.492276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.493074 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.493078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.493229 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.493246 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.493314 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.493409 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":73,"thread_start_us":120,"threads_started":1}
I20260929 18:38:22.493593 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.489213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39505 (local address 127.0.198.2:60443)
0929 18:38:22.489327 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.489343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.489454 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.489834 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.489836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.489855 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:22.490000 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.490008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.490963 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.490965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.492217 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.492228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.493074 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.493078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.493212 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.493243 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.493319 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.493409 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":4}
I20260929 18:38:22.494195 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7ab6960c884dbdb98d871ec9ce202c" candidate_uuid: "c7a40c8f558a468e87fe0e6438b8d424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456b28faff91441598be5a2d79886ef6" is_pre_election: true
I20260929 18:38:22.494390 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7ab6960c884dbdb98d871ec9ce202c" candidate_uuid: "c7a40c8f558a468e87fe0e6438b8d424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" is_pre_election: true
I20260929 18:38:22.494524 14714 raft_consensus.cc:2454] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a40c8f558a468e87fe0e6438b8d424 in term 0.
I20260929 18:38:22.494622 14982 raft_consensus.cc:2454] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a40c8f558a468e87fe0e6438b8d424 in term 0.
I20260929 18:38:22.494961 14784 leader_election.cc:302] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456b28faff91441598be5a2d79886ef6, c7a40c8f558a468e87fe0e6438b8d424; no voters:
I20260929 18:38:22.495224 15052 raft_consensus.cc:2781] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:38:22.495308 15052 raft_consensus.cc:482] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:38:22.495349 15052 raft_consensus.cc:3037] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:22.496321 15052 raft_consensus.cc:504] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.496619 15052 leader_election.cc:288] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [CANDIDATE]: Term 1 election: Requested vote from peers c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505), 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225)
I20260929 18:38:22.496930 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7ab6960c884dbdb98d871ec9ce202c" candidate_uuid: "c7a40c8f558a468e87fe0e6438b8d424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a43de10c0b45aebf1284e1f1b41a40"
I20260929 18:38:22.496981 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7ab6960c884dbdb98d871ec9ce202c" candidate_uuid: "c7a40c8f558a468e87fe0e6438b8d424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456b28faff91441598be5a2d79886ef6"
I20260929 18:38:22.497051 14982 raft_consensus.cc:3037] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:22.497093 14714 raft_consensus.cc:3037] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:22.498327 14714 raft_consensus.cc:2454] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a40c8f558a468e87fe0e6438b8d424 in term 1.
I20260929 18:38:22.498329 14982 raft_consensus.cc:2454] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a40c8f558a468e87fe0e6438b8d424 in term 1.
I20260929 18:38:22.498620 14784 leader_election.cc:302] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456b28faff91441598be5a2d79886ef6, c7a40c8f558a468e87fe0e6438b8d424; no voters:
I20260929 18:38:22.498790 15052 raft_consensus.cc:2781] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:22.499014 15052 raft_consensus.cc:686] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 LEADER]: Becoming Leader. State: Replica: c7a40c8f558a468e87fe0e6438b8d424, State: Running, Role: LEADER
I20260929 18:38:22.499318 15052 consensus_queue.cc:237] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.502534 14569 catalog_manager.cc:5495] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "c7a40c8f558a468e87fe0e6438b8d424" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } health_report { overall_health: UNKNOWN } } }
I20260929 18:38:22.527056 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.523577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.0.198.2:35537)
0929 18:38:22.523705 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.523709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.523723 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:38:22.523854 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.523856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.523914 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.524004 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:38:22.524221 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.524722 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.525609 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.525821 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.525905 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.526246 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:38:22.526663 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:38:22.526768 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.526839 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.526895 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:38:22.533016 14714 raft_consensus.cc:1261] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Refusing update from remote peer c7a40c8f558a468e87fe0e6438b8d424: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:22.532998 14982 raft_consensus.cc:1261] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 FOLLOWER]: Refusing update from remote peer c7a40c8f558a468e87fe0e6438b8d424: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:22.533793 15055 consensus_queue.cc:1035] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:22.534014 15052 consensus_queue.cc:1035] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:22.542204 15059 mvcc.cc:205] Tried to move back new op lower bound from 7334736291967533056 to 7334736291839758336. Current Snapshot: MvccSnapshot[applied={T|T < 7334736291967533056}]
I20260929 18:38:22.550244 14568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:22.550410 14568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:22.552795 14568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Sending DeleteTablet for 3 replicas of tablet cf7ab6960c884dbdb98d871ec9ce202c
I20260929 18:38:22.553597 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14631
I20260929 18:38:22.553684 14694 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:22 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:36700
I20260929 18:38:22.553735 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:22 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:47244
I20260929 18:38:22.555208 15068 tablet_replica.cc:330] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: stopping tablet replica
I20260929 18:38:22.555338 14828 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:22 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:44784
I20260929 18:38:22.555588 15068 raft_consensus.cc:2229] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:22.555883 15068 raft_consensus.cc:2258] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:22.557446 15068 ts_tablet_manager.cc:1905] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:38:22.559530 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:41225 (error 108)
I20260929 18:38:22.559541 15069 tablet_replica.cc:330] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: stopping tablet replica
I20260929 18:38:22.559862 15069 raft_consensus.cc:2229] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:38:22.560233 15068 ts_tablet_manager.cc:1918] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:22.560243 15069 raft_consensus.cc:2258] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:22.560369 15068 log.cc:1196] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/cf7ab6960c884dbdb98d871ec9ce202c
I20260929 18:38:22.560655 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14765
I20260929 18:38:22.562907 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.562635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41225 (local address 127.0.0.1:36702)
0929 18:38:22.562766 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.562829 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 18:38:22.563063 14556 catalog_manager.cc:4584] TS 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf7ab6960c884dbdb98d871ec9ce202c: Network error: Client connection negotiation failed: client connection to 127.0.198.1:41225: connect: Connection refused (error 111)
I20260929 18:38:22.565611 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14899
I20260929 18:38:22.565651 15068 ts_tablet_manager.cc:1939] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Deleting consensus metadata
I20260929 18:38:22.566923 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.566689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35537 (local address 127.0.0.1:44810)
0929 18:38:22.566806 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.566853 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.2:35537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:38:22.570816 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 14537
I20260929 18:38:22.575990 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41069
--webserver_interface=127.0.198.62
--webserver_port=43071
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:22.685205 15071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:22.685487 15071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:22.685526 15071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:22.689618 15071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:22.689684 15071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:22.689702 15071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:22.689718 15071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:22.689731 15071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:22.694988 15071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:22.696020 15071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:22.697003 15071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:22.709376 15071 server_base.cc:1034] running on GCE node
W20260929 18:38:22.709311 15078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:22.709311 15080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:22.709331 15077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:22.710300 15071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:22.711464 15071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:22.712652 15071 hybrid_clock.cc:648] HybridClock initialized: now 1790707102712630 us; error 35 us; skew 500 ppm
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:22.715771 15071 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:22.716693 15071 webserver.cc:460] Webserver started at http://127.0.198.62:43071/ using document root <none> and password file <none>
I20260929 18:38:22.717257 15071 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:22.717338 15071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:22.721251 15071 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:22.723414 15087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.724715 15071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:38:22.724851 15071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "62fe08d8657f4356b1de6e338d4cd3a3"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "bc35921113f5c93afe0f7c3e77ec3cf5"
server_key_iv: "58fab3a55c53e52dfb40946abebd76d8"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.725445 15071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:22.742430 15071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:22.745643 15071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:22.745834 15071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:22.752637 15071 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:41069
I20260929 18:38:22.752940 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:41069 every 8 connection(s)
I20260929 18:38:22.753726 15071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:22.758157 15140 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:38:22.759783 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Bootstrap starting.
I20260929 18:38:22.762615 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15071
I20260929 18:38:22.762997 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:41225
--local_ip_for_outbound_sockets=127.0.198.1
--tserver_master_addrs=127.0.198.62:41069
--webserver_port=37119
--webserver_interface=127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:22.776844 15140 log.cc:824] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:22.785552 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:38:22.785899 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Bootstrap complete.
I20260929 18:38:22.789693 15140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:22.790127 15140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fe08d8657f4356b1de6e338d4cd3a3, State: Initialized, Role: FOLLOWER
I20260929 18:38:22.790633 15140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:22.790753 15140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:22.790825 15140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:22.790920 15140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:38:22.792749 15140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:22.793076 15140 leader_election.cc:302] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62fe08d8657f4356b1de6e338d4cd3a3; no voters:
I20260929 18:38:22.793361 15140 leader_election.cc:288] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:38:22.793460 15144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:38:22.793656 15144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 LEADER]: Becoming Leader. State: Replica: 62fe08d8657f4356b1de6e338d4cd3a3, State: Running, Role: LEADER
I20260929 18:38:22.794044 15144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:22.794665 15140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:22.795470 15145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } } }
I20260929 18:38:22.795521 15146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fe08d8657f4356b1de6e338d4cd3a3. Latest consensus state: current_term: 2 leader_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } } }
I20260929 18:38:22.795637 15146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:22.795591 15145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:22.799932 15159 catalog_manager.cc:1259] Loaded cluster ID: 4b827cb07bc044db9111dfa34d3078e6
I20260929 18:38:22.799999 15159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: loading cluster ID for follower catalog manager: success
W20260929 18:38:22.801011 15159 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260929 18:38:22.801084 15159 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260929 18:38:22.801146 15153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:22.803592 15153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:22.804049 15153 catalog_manager.cc:1259] Loaded cluster ID: 4b827cb07bc044db9111dfa34d3078e6
I20260929 18:38:22.804091 15153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:22.805080 15153 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 18:38:22.809427 15153 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 18:38:22.814843 15153 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:22.815641 15153 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Loaded TSK: 0
I20260929 18:38:22.816834 15153 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:38:22.881911 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:22.882233 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:22.882277 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:22.886409 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:22.886477 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:22.886547 15142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:38:22.891920 15142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:41225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=37119
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:22.892968 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:22.894011 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:22.903818 15167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:22.903815 15170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:22.903815 15168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:22.904407 15142 server_base.cc:1034] running on GCE node
I20260929 18:38:22.904824 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:22.905596 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:22.906831 15142 hybrid_clock.cc:648] HybridClock initialized: now 1790707102906795 us; error 48 us; skew 500 ppm
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:22.909848 15142 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:38:22.910908 15142 webserver.cc:460] Webserver started at http://127.0.198.1:37119/ using document root <none> and password file <none>
I20260929 18:38:22.911453 15142 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:22.911532 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:22.915690 15142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:38:22.917822 15177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:22.919147 15142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:22.919263 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "456b28faff91441598be5a2d79886ef6"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "f612f50ebe1ebbf0c6364a484a39173c"
server_key_iv: "3c2a251abee3b65f0e704b68fa99c075"
server_key_version: "encryptionkey@0"
I20260929 18:38:22.919842 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:22.950923 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:22.954012 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:22.954178 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:22.954782 15142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:22.955852 15184 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:38:22.960063 15142 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:38:22.960134 15142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 18:38:22.960191 15142 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:38:22.961920 15142 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:38:22.961998 15142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:22.962255 15184 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Bootstrap starting.
I20260929 18:38:22.971612 15142 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:41225
I20260929 18:38:22.971693 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:41225 every 8 connection(s)
I20260929 18:38:22.972586 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:22.979601 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15142
I20260929 18:38:22.979674 15184 log.cc:824] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:22.980007 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:35537
--local_ip_for_outbound_sockets=127.0.198.2
--tserver_master_addrs=127.0.198.62:41069
--webserver_port=40989
--webserver_interface=127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:22.988114 15184 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:38:22.988545 15184 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Bootstrap complete.
I20260929 18:38:22.989383 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.974580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:40295 (local address 127.0.198.62:41069)
0929 18:38:22.974999 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.975020 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.975298 (+ 278us) server_negotiation.cc:407] Connection header received
0929 18:38:22.976447 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.976463 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.976883 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.977325 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:22.978444 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.979918 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.981686 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.982232 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.982392 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.984699 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:22.984704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:22.984716 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:22.984743 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:22.986317 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.986797 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.986802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.986806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.986918 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:22.987222 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:22.987225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:22.987227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:22.987580 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:22.987696 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:22.988225 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 18:38:22.988508 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":121,"threads_started":1}
I20260929 18:38:22.989394 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.974528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:40295)
0929 18:38:22.975125 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.975178 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.975992 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.977364 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.977372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.977754 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:22.978268 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.978283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.980280 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.980284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.981563 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.981574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:22.982362 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:22.982366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:22.982556 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:22.983086 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:22.983105 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:22.984511 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:22.986427 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:22.986433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:22.986447 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:22.986689 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:22.987006 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:22.987009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:22.987011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:22.987137 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:22.987688 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:22.987693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:22.987932 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 18:38:22.988097 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 18:38:22.988512 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":150,"threads_started":1}
I20260929 18:38:22.989845 15184 ts_tablet_manager.cc:1397] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.007s
I20260929 18:38:22.991979 15292 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41069
I20260929 18:38:22.992188 15292 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:22.992628 15292 heartbeater.cc:511] Master 127.0.198.62:41069 requested a full tablet report, sending...
I20260929 18:38:22.992938 15184 raft_consensus.cc:348] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.993441 15184 raft_consensus.cc:729] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456b28faff91441598be5a2d79886ef6, State: Initialized, Role: FOLLOWER
I20260929 18:38:22.993916 15184 consensus_queue.cc:260] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:22.994894 15184 ts_tablet_manager.cc:1428] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:38:22.994966 15104 ts_manager.cc:194] Registered new tserver with Master: 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225)
I20260929 18:38:22.998265 15104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:40295
I20260929 18:38:23.000710 15292 heartbeater.cc:503] Master 127.0.198.62:41069 was elected leader, sending a full tablet report...
Sep 29 18:38:22 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:38:23.008488 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.997810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41225 (local address 127.0.0.1:36712)
0929 18:38:22.998222 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:22.998266 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:22.998476 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:22.999337 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:22.999343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:22.999778 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:22.999918 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:22.999926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.001155 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.001159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.002171 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.002182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.002767 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.002771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.002880 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.003525 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:23.003540 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:23.005048 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:23.006814 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:23.006824 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:23.006830 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:23.007050 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:23.007326 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:23.007329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:23.007331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:23.007411 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:23.007848 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:23.007854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:23.007973 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 18:38:23.008091 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:38:23.008290 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260929 18:38:23.008488 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:22.998590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.0.198.1:41225)
0929 18:38:22.998883 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:22.998894 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:22.998917 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:38:22.998984 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:22.998989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:22.999187 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:22.999320 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:23.000044 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.001032 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.002274 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.002658 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.002741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.005182 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:23.005193 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:23.005204 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:23.005225 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:23.006704 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.007157 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.007163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.007164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.007240 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.007497 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.007500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.007502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.007739 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:23.007854 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.008184 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.008341 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260929 18:38:23.009472 15226 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:22 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:36712
I20260929 18:38:23.010335 15301 tablet_replica.cc:330] stopping tablet replica
I20260929 18:38:23.010753 15301 raft_consensus.cc:2229] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:23.011045 15301 raft_consensus.cc:2258] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:23.012125 15301 ts_tablet_manager.cc:1905] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:23.016775 15301 ts_tablet_manager.cc:1918] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:23.016887 15301 log.cc:1196] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/cf7ab6960c884dbdb98d871ec9ce202c
I20260929 18:38:23.017162 15301 ts_tablet_manager.cc:1939] T cf7ab6960c884dbdb98d871ec9ce202c P 456b28faff91441598be5a2d79886ef6: Deleting consensus metadata
I20260929 18:38:23.019507 15091 catalog_manager.cc:4841] TS 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225): tablet cf7ab6960c884dbdb98d871ec9ce202c (table test-table [id=243d916bb25f49fe92b1c09b8589889f]) successfully deleted
W20260929 18:38:23.100314 15297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:23.100610 15297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:23.100659 15297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:23.104823 15297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:23.104887 15297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:23.104948 15297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:38:23.110183 15297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:35537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=40989
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:23.111238 15297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:23.112224 15297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:23.122705 15297 server_base.cc:1034] running on GCE node
W20260929 18:38:23.122656 15306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:23.122663 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:23.122715 15307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:23.123518 15297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:23.124189 15297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:23.125347 15297 hybrid_clock.cc:648] HybridClock initialized: now 1790707103125330 us; error 42 us; skew 500 ppm
Sep 29 18:38:23 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:23.128018 15297 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:38:23.128909 15297 webserver.cc:460] Webserver started at http://127.0.198.2:40989/ using document root <none> and password file <none>
I20260929 18:38:23.129462 15297 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:23.129549 15297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:23.133287 15297 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 18:38:23.135519 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:23.136829 15297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:38:23.136943 15297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "c7a40c8f558a468e87fe0e6438b8d424"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "0348692a8ac11d0515af282f26449ad5"
server_key_iv: "9edcd1b9930eeae594939185cd2bc1dc"
server_key_version: "encryptionkey@0"
I20260929 18:38:23.137493 15297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:23.160715 15297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:23.163553 15297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:23.163733 15297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:23.164309 15297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:23.165377 15323 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:38:23.169456 15297 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:38:23.169540 15297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 18:38:23.169625 15297 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:38:23.171255 15297 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:38:23.171300 15297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 18:38:23.171566 15323 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Bootstrap starting.
I20260929 18:38:23.181030 15297 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:35537
I20260929 18:38:23.181079 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:35537 every 8 connection(s)
I20260929 18:38:23.181960 15297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:23.189028 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15297
I20260929 18:38:23.189476 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:39505
--local_ip_for_outbound_sockets=127.0.198.3
--tserver_master_addrs=127.0.198.62:41069
--webserver_port=41863
--webserver_interface=127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:23.189754 15323 log.cc:824] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:38:23 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:23.197405 15323 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:38:23.197552 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.184133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:50203 (local address 127.0.198.62:41069)
0929 18:38:23.184286 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.184290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.184999 (+ 709us) server_negotiation.cc:407] Connection header received
0929 18:38:23.186224 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.186227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.186280 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.186353 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:23.187943 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.188515 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.190142 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.190470 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.190581 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.193298 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:23.193302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:23.193304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:23.193336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:23.195163 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.195769 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.195773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.195775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.195879 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.196312 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.196315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.196317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.196571 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:23.196678 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.197234 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.197376 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:38:23.197801 15323 tablet_bootstrap.cc:492] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Bootstrap complete.
I20260929 18:38:23.198544 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.184254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:50203)
0929 18:38:23.184814 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:23.184878 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:23.185936 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:23.186533 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:23.186542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:23.187034 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:23.187780 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.187800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.188677 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.188681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.190000 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.190011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.190612 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.190616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.190803 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.191466 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:23.191486 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:23.193112 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:23.195270 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:23.195275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:23.195287 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:23.195647 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:23.195983 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:23.195986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:23.195988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:23.196223 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:23.196680 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:23.196685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:23.196968 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 18:38:23.197171 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 18:38:23.197489 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":135,"threads_started":1}
I20260929 18:38:23.199003 15323 ts_tablet_manager.cc:1397] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.013s
I20260929 18:38:23.199723 15431 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41069
I20260929 18:38:23.199940 15431 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:23.200467 15431 heartbeater.cc:511] Master 127.0.198.62:41069 requested a full tablet report, sending...
I20260929 18:38:23.201856 15323 raft_consensus.cc:348] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.202303 15104 ts_manager.cc:194] Registered new tserver with Master: c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537)
I20260929 18:38:23.202379 15323 raft_consensus.cc:729] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a40c8f558a468e87fe0e6438b8d424, State: Initialized, Role: FOLLOWER
I20260929 18:38:23.202867 15323 consensus_queue.cc:260] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.203578 15104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:50203
I20260929 18:38:23.203898 15323 ts_tablet_manager.cc:1428] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:38:23.206084 15431 heartbeater.cc:503] Master 127.0.198.62:41069 was elected leader, sending a full tablet report...
I20260929 18:38:23.209288 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.204529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.0.198.2:35537)
0929 18:38:23.204831 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.204842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.204862 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:38:23.204927 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.204932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.205231 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.205384 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:23.205637 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.206604 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.208199 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.208759 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.208845 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.208941 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.209125 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.209193 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":123,"threads_started":1}
I20260929 18:38:23.209313 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.204046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35537 (local address 127.0.0.1:44814)
0929 18:38:23.204209 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:23.204224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:23.204296 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:23.205396 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:23.205400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:23.205422 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:23.205522 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.205528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.206728 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.206732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.208075 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.208086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.208871 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.208874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.208975 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.209031 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 18:38:23.209102 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:38:23.209185 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 18:38:23.210417 15365 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7ab6960c884dbdb98d871ec9ce202c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:22 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:44814
I20260929 18:38:23.211251 15438 tablet_replica.cc:330] stopping tablet replica
I20260929 18:38:23.211567 15438 raft_consensus.cc:2229] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:23.211843 15438 raft_consensus.cc:2258] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:23.212914 15438 ts_tablet_manager.cc:1905] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:23.217538 15438 ts_tablet_manager.cc:1918] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:23.217645 15438 log.cc:1196] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/cf7ab6960c884dbdb98d871ec9ce202c
I20260929 18:38:23.217933 15438 ts_tablet_manager.cc:1939] T cf7ab6960c884dbdb98d871ec9ce202c P c7a40c8f558a468e87fe0e6438b8d424: Deleting consensus metadata
I20260929 18:38:23.219820 15091 catalog_manager.cc:4841] TS c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537): tablet cf7ab6960c884dbdb98d871ec9ce202c (table test-table [id=243d916bb25f49fe92b1c09b8589889f]) successfully deleted
W20260929 18:38:23.308845 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:23.309139 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:23.309182 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:23.313447 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:23.313515 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:23.313580 15435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:38:23.319135 15435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:39505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=41863
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:23.320187 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:23.321169 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:23.331192 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:23.331369 15435 server_base.cc:1034] running on GCE node
W20260929 18:38:23.331187 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:23.331187 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:23.331972 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:23.332623 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:23.333792 15435 hybrid_clock.cc:648] HybridClock initialized: now 1790707103333753 us; error 57 us; skew 500 ppm
Sep 29 18:38:23 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:23.336668 15435 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:38:23.337591 15435 webserver.cc:460] Webserver started at http://127.0.198.3:41863/ using document root <none> and password file <none>
I20260929 18:38:23.338178 15435 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:23.338263 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:23.342447 15435 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:38:23.344589 15453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:23.345902 15435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:23.346051 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "c3a43de10c0b45aebf1284e1f1b41a40"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "c20f0c281847c500ff9066f1cab0bc7b"
server_key_iv: "c4db0e6e25fb98dea892815b6c24c77c"
server_key_version: "encryptionkey@0"
I20260929 18:38:23.346660 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:23.377094 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:23.380168 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:23.380352 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:23.380934 15435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:23.381939 15460 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 18:38:23.385707 15460 ts_tablet_manager.cc:1852] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 18:38:23.386127 15460 ts_tablet_manager.cc:1905] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:23.391443 15460 ts_tablet_manager.cc:1918] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:23.391660 15460 ts_tablet_manager.cc:1939] T cf7ab6960c884dbdb98d871ec9ce202c P c3a43de10c0b45aebf1284e1f1b41a40: Deleting consensus metadata
I20260929 18:38:23.392032 15435 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 18:38:23.392091 15435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260929 18:38:23.392159 15435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:23.392189 15435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:23.401510 15435 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:39505
I20260929 18:38:23.401569 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:39505 every 8 connection(s)
I20260929 18:38:23.402432 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:23.407621 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15435
Sep 29 18:38:23 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:23.416560 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.408392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.0.198.62:41069)
0929 18:38:23.408719 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.408723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.408818 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:38:23.408942 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.408945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.408989 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.409057 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:23.409917 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.410592 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.411649 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.411879 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.411965 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.413280 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:23.413283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:23.413285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:23.413307 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:23.414816 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.415272 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.415275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.415276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.415345 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.415622 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.415625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.415626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.415880 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:23.416027 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.416235 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.416362 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"spinlock_wait_cycles":1920,"thread_start_us":152,"threads_started":1}
I20260929 18:38:23.417196 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.404235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:53061 (local address 127.0.198.62:41069)
0929 18:38:23.404400 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.404405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.405280 (+ 875us) server_negotiation.cc:407] Connection header received
0929 18:38:23.406217 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.406220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.406268 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.406348 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:23.407895 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.408688 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.410300 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.410613 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.410700 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.413605 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:23.413607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:23.413609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:23.413642 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:23.414986 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.415607 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.415612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.415613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.415744 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:23.416083 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:23.416086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:23.416088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:23.416260 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:23.416348 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.416935 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.417056 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":2944}
I20260929 18:38:23.418219 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.404646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:53061)
0929 18:38:23.405120 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:23.405162 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:23.406000 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:23.406672 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:23.406679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:23.407041 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:23.407655 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.407677 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.408851 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.408854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.410195 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.410205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.410760 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.410764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.410963 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.411645 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:23.411666 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:23.413422 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:23.415084 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:23.415090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:23.415108 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:23.415413 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:23.415832 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:23.415834 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:23.415836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:23.415989 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:23.416370 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:23.416376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:23.416674 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 18:38:23.416879 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 18:38:23.417216 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260929 18:38:23.419284 15568 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41069
I20260929 18:38:23.419551 15568 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:23.420027 15568 heartbeater.cc:511] Master 127.0.198.62:41069 requested a full tablet report, sending...
I20260929 18:38:23.420951 15104 ts_manager.cc:194] Registered new tserver with Master: c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505)
I20260929 18:38:23.421468 15104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:53061
I20260929 18:38:23.430052 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:38:23.434198 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.430627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.0.198.62:41069)
0929 18:38:23.430754 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.430758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.430884 (+ 126us) server_negotiation.cc:407] Connection header received
0929 18:38:23.431033 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.431035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.431138 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.431218 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:38:23.431450 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.431950 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.432849 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.433080 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.433158 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.433428 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:38:23.433819 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:38:23.433906 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.434049 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.434098 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:38:23.434789 15104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:38:23.437911 15104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:38:23.446385 15365 tablet_service.cc:1467] Processing CreateTablet for tablet ae1ae1980ffb4a329bd9d143a8802508 (DEFAULT_TABLE table=test-table [id=ab484013b8b44e92b62dc6926488b7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:23.446417 15226 tablet_service.cc:1467] Processing CreateTablet for tablet ae1ae1980ffb4a329bd9d143a8802508 (DEFAULT_TABLE table=test-table [id=ab484013b8b44e92b62dc6926488b7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:23.447041 15365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1ae1980ffb4a329bd9d143a8802508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:23.447038 15226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1ae1980ffb4a329bd9d143a8802508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:23.449740 15323 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: Bootstrap starting.
I20260929 18:38:23.450235 15184 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Bootstrap starting.
I20260929 18:38:23.450666 15323 tablet_bootstrap.cc:654] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:23.450953 15184 tablet_bootstrap.cc:654] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:23.451687 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.445529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39505 (local address 127.0.0.1:47256)
0929 18:38:23.445707 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:23.445719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:23.445803 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:23.447077 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:23.447081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:23.447101 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:23.447205 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.447212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.448930 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.448934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.450040 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.450053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.451189 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.451194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.451302 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.451332 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:38:23.451417 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:38:23.451555 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 18:38:23.451754 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.446010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.0.198.3:39505)
0929 18:38:23.446498 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.446511 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.446536 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:38:23.446618 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.446624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.446899 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.447370 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:23.447677 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.448782 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.450167 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.451060 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.451182 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.451339 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.451452 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.451599 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":164,"threads_started":1}
I20260929 18:38:23.452354 15323 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: No bootstrap required, opened a new log
I20260929 18:38:23.452440 15323 ts_tablet_manager.cc:1397] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:38:23.452556 15184 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: No bootstrap required, opened a new log
I20260929 18:38:23.452639 15184 ts_tablet_manager.cc:1397] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:38:23.452880 15323 raft_consensus.cc:348] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.453001 15323 raft_consensus.cc:374] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:23.453034 15323 raft_consensus.cc:729] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a40c8f558a468e87fe0e6438b8d424, State: Initialized, Role: FOLLOWER
I20260929 18:38:23.453025 15184 raft_consensus.cc:348] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.453109 15184 raft_consensus.cc:374] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:23.453166 15184 raft_consensus.cc:729] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456b28faff91441598be5a2d79886ef6, State: Initialized, Role: FOLLOWER
I20260929 18:38:23.453156 15323 consensus_queue.cc:260] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.453287 15184 consensus_queue.cc:260] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.453387 15323 ts_tablet_manager.cc:1428] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:38:23.453516 15184 ts_tablet_manager.cc:1428] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:38:23.453549 15502 tablet_service.cc:1467] Processing CreateTablet for tablet ae1ae1980ffb4a329bd9d143a8802508 (DEFAULT_TABLE table=test-table [id=ab484013b8b44e92b62dc6926488b7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:23.454604 15502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1ae1980ffb4a329bd9d143a8802508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:23.458532 15460 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Bootstrap starting.
I20260929 18:38:23.460700 15460 tablet_bootstrap.cc:654] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:23.461395 15460 log.cc:824] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:23.463246 15460 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: No bootstrap required, opened a new log
I20260929 18:38:23.463428 15460 ts_tablet_manager.cc:1397] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260929 18:38:23.466379 15460 raft_consensus.cc:348] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.466598 15460 raft_consensus.cc:374] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:23.466645 15460 raft_consensus.cc:729] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a43de10c0b45aebf1284e1f1b41a40, State: Initialized, Role: FOLLOWER
I20260929 18:38:23.467039 15460 consensus_queue.cc:260] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.467675 15568 heartbeater.cc:503] Master 127.0.198.62:41069 was elected leader, sending a full tablet report...
I20260929 18:38:23.467973 15460 ts_tablet_manager.cc:1428] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260929 18:38:23.473754 15293 tablet.cc:2369] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:38:23.545403 15581 raft_consensus.cc:482] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:23.545632 15581 raft_consensus.cc:504] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.546844 15581 leader_election.cc:288] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537), 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225)
I20260929 18:38:23.551107 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.547208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:58711 (local address 127.0.198.2:35537)
0929 18:38:23.547350 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.547355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.547373 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:38:23.547584 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.547587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.547640 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.547722 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:23.548010 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.548625 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.549949 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.550566 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.550658 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.550720 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.550926 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.551008 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:38:23.551117 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.547125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35537 (local address 127.0.198.3:58711)
0929 18:38:23.547273 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:23.547291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:23.547404 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:23.547746 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:23.547750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:23.547772 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:23.547885 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.547892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.548729 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.548732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.549828 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.549837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.550680 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.550684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.550799 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.550825 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:38:23.550911 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:38:23.550984 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:38:23.551779 15385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1ae1980ffb4a329bd9d143a8802508" candidate_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a40c8f558a468e87fe0e6438b8d424" is_pre_election: true
I20260929 18:38:23.551891 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.547510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:47751 (local address 127.0.198.1:41225)
0929 18:38:23.547745 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.547750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.547836 (+ 86us) server_negotiation.cc:407] Connection header received
0929 18:38:23.547944 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.547947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.548022 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.548105 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:23.548360 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.548914 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.550656 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.551399 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.551501 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.551633 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.551742 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.551797 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":54,"threads_started":1}
I20260929 18:38:23.551920 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.547380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41225 (local address 127.0.198.3:47751)
0929 18:38:23.547757 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:23.547770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:23.547858 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:23.548131 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:23.548133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:23.548151 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:23.548260 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.548266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.549059 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.549062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.550510 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:23.550521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.551531 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.551535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:23.551628 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.551651 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:38:23.551728 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:38:23.551795 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":136,"threads_started":1}
I20260929 18:38:23.552124 15385 raft_consensus.cc:2454] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a43de10c0b45aebf1284e1f1b41a40 in term 0.
I20260929 18:38:23.552563 15457 leader_election.cc:302] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a43de10c0b45aebf1284e1f1b41a40, c7a40c8f558a468e87fe0e6438b8d424; no voters:
I20260929 18:38:23.552577 15246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1ae1980ffb4a329bd9d143a8802508" candidate_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456b28faff91441598be5a2d79886ef6" is_pre_election: true
I20260929 18:38:23.552824 15246 raft_consensus.cc:2454] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a43de10c0b45aebf1284e1f1b41a40 in term 0.
I20260929 18:38:23.552850 15581 raft_consensus.cc:2781] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:38:23.552929 15581 raft_consensus.cc:482] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:38:23.552989 15581 raft_consensus.cc:3037] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:23.554065 15581 raft_consensus.cc:504] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.554363 15581 leader_election.cc:288] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [CANDIDATE]: Term 1 election: Requested vote from peers c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537), 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225)
I20260929 18:38:23.554689 15385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1ae1980ffb4a329bd9d143a8802508" candidate_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a40c8f558a468e87fe0e6438b8d424"
I20260929 18:38:23.554760 15246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1ae1980ffb4a329bd9d143a8802508" candidate_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456b28faff91441598be5a2d79886ef6"
I20260929 18:38:23.554819 15385 raft_consensus.cc:3037] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:23.554869 15246 raft_consensus.cc:3037] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:23.556145 15246 raft_consensus.cc:2454] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a43de10c0b45aebf1284e1f1b41a40 in term 1.
I20260929 18:38:23.556332 15385 raft_consensus.cc:2454] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a43de10c0b45aebf1284e1f1b41a40 in term 1.
I20260929 18:38:23.556480 15457 leader_election.cc:302] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456b28faff91441598be5a2d79886ef6, c3a43de10c0b45aebf1284e1f1b41a40; no voters:
I20260929 18:38:23.556644 15581 raft_consensus.cc:2781] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:23.556838 15581 raft_consensus.cc:686] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 LEADER]: Becoming Leader. State: Replica: c3a43de10c0b45aebf1284e1f1b41a40, State: Running, Role: LEADER
I20260929 18:38:23.557157 15581 consensus_queue.cc:237] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:23.560381 15101 catalog_manager.cc:5495] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3). New cstate: current_term: 1 leader_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } health_report { overall_health: UNKNOWN } } }
I20260929 18:38:23.582775 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:23.579257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.0.198.3:39505)
0929 18:38:23.579391 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:23.579395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:23.579408 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:23.579509 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:23.579511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:23.579566 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:23.579655 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:38:23.579906 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.580403 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.581297 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:23.581523 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:23.581611 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:23.581920 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:38:23.582387 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:38:23.582507 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:23.582572 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:38:23.582633 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:38:23.589064 15385 raft_consensus.cc:1261] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Refusing update from remote peer c3a43de10c0b45aebf1284e1f1b41a40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:23.589064 15246 raft_consensus.cc:1261] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Refusing update from remote peer c3a43de10c0b45aebf1284e1f1b41a40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:23.589857 15584 consensus_queue.cc:1035] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:23.590085 15581 consensus_queue.cc:1035] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:23.603896 15101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:23.604105 15101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:23.606804 15101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Sending DeleteTablet for 3 replicas of tablet ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:23.607599 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 15142
I20260929 18:38:23.607592 15365 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1ae1980ffb4a329bd9d143a8802508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:23 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:44814
I20260929 18:38:23.607641 15226 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1ae1980ffb4a329bd9d143a8802508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:23 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:36712
I20260929 18:38:23.607784 15438 tablet_replica.cc:330] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: stopping tablet replica
I20260929 18:38:23.607883 15438 raft_consensus.cc:2229] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:23.608021 15438 raft_consensus.cc:2258] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:23.608773 15502 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1ae1980ffb4a329bd9d143a8802508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:23 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:47256
I20260929 18:38:23.609517 15438 ts_tablet_manager.cc:1905] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:23.610184 15597 tablet_replica.cc:330] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: stopping tablet replica
I20260929 18:38:23.610574 15597 raft_consensus.cc:2229] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:38:23.611055 15597 raft_consensus.cc:2258] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:23.612205 15438 ts_tablet_manager.cc:1918] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:23.612287 15438 log.cc:1196] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:23.612535 15438 ts_tablet_manager.cc:1939] T ae1ae1980ffb4a329bd9d143a8802508 P c7a40c8f558a468e87fe0e6438b8d424: Deleting consensus metadata
I20260929 18:38:23.613371 15091 catalog_manager.cc:4841] TS c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537): tablet ae1ae1980ffb4a329bd9d143a8802508 (table test-table [id=ab484013b8b44e92b62dc6926488b7a6]) successfully deleted
W20260929 18:38:23.614012 15091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:41225 (error 108)
I20260929 18:38:23.614382 15597 ts_tablet_manager.cc:1905] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:23.615134 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 15297
W20260929 18:38:23.615154 15299 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:23.614811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:41225 (local address 127.0.0.1:36728)
0929 18:38:23.614975 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:23.615057 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:41225: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 18:38:23.615396 15091 catalog_manager.cc:4584] TS 456b28faff91441598be5a2d79886ef6 (127.0.198.1:41225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae1ae1980ffb4a329bd9d143a8802508: Network error: Client connection negotiation failed: client connection to 127.0.198.1:41225: connect: Connection reset by peer (error 104)
I20260929 18:38:23.617022 15597 ts_tablet_manager.cc:1918] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:23.617127 15597 log.cc:1196] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:23.617440 15597 ts_tablet_manager.cc:1939] T ae1ae1980ffb4a329bd9d143a8802508 P c3a43de10c0b45aebf1284e1f1b41a40: Deleting consensus metadata
I20260929 18:38:23.618851 15091 catalog_manager.cc:4841] TS c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505): tablet ae1ae1980ffb4a329bd9d143a8802508 (table test-table [id=ab484013b8b44e92b62dc6926488b7a6]) successfully deleted
I20260929 18:38:23.621052 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 15435
I20260929 18:38:23.626129 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 15071
I20260929 18:38:23.631636 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41069
--webserver_interface=127.0.198.62
--webserver_port=43071
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:23.740038 15599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:23.740309 15599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:23.740370 15599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:23.744356 15599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:23.744418 15599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:23.744444 15599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:23.744464 15599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:23.744493 15599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:23.749832 15599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:23.750927 15599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:23.751905 15599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:23.760206 15604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:23.760123 15607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:23.760105 15605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:23.760398 15599 server_base.cc:1034] running on GCE node
I20260929 18:38:23.760967 15599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:23.762138 15599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:23.763312 15599 hybrid_clock.cc:648] HybridClock initialized: now 1790707103763293 us; error 33 us; skew 500 ppm
Sep 29 18:38:23 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:23.766324 15599 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:23.767197 15599 webserver.cc:460] Webserver started at http://127.0.198.62:43071/ using document root <none> and password file <none>
I20260929 18:38:23.767716 15599 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:23.767782 15599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:23.771440 15599 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 18:38:23.773423 15614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:23.774704 15599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:38:23.774824 15599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "62fe08d8657f4356b1de6e338d4cd3a3"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "bc35921113f5c93afe0f7c3e77ec3cf5"
server_key_iv: "58fab3a55c53e52dfb40946abebd76d8"
server_key_version: "encryptionkey@0"
I20260929 18:38:23.775354 15599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:23.794549 15599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:23.797883 15599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:23.798115 15599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:23.805104 15599 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:41069
I20260929 18:38:23.805425 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:41069 every 8 connection(s)
I20260929 18:38:23.806212 15599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:23.807888 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15599
I20260929 18:38:23.808286 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:41225
--local_ip_for_outbound_sockets=127.0.198.1
--tserver_master_addrs=127.0.198.62:41069
--webserver_port=37119
--webserver_interface=127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:23.810839 15667 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:38:23.812562 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Bootstrap starting.
I20260929 18:38:23.830610 15667 log.cc:824] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:23.842263 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:38:23.842645 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3: Bootstrap complete.
I20260929 18:38:23.846534 15667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:23.846940 15667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fe08d8657f4356b1de6e338d4cd3a3, State: Initialized, Role: FOLLOWER
I20260929 18:38:23.847414 15667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:23.847537 15667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:23.847608 15667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:23.847698 15667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:38:23.849766 15667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:23.850128 15667 leader_election.cc:302] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62fe08d8657f4356b1de6e338d4cd3a3; no voters:
I20260929 18:38:23.850440 15667 leader_election.cc:288] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:38:23.850553 15671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:38:23.850736 15671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [term 3 LEADER]: Becoming Leader. State: Replica: 62fe08d8657f4356b1de6e338d4cd3a3, State: Running, Role: LEADER
I20260929 18:38:23.851023 15671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } }
I20260929 18:38:23.851682 15667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:23.851989 15671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fe08d8657f4356b1de6e338d4cd3a3. Latest consensus state: current_term: 3 leader_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } } }
I20260929 18:38:23.852280 15671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:23.853173 15676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:23.853447 15672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fe08d8657f4356b1de6e338d4cd3a3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41069 } } }
I20260929 18:38:23.853626 15672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fe08d8657f4356b1de6e338d4cd3a3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:23.858240 15676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:23.859215 15676 catalog_manager.cc:1259] Loaded cluster ID: 4b827cb07bc044db9111dfa34d3078e6
I20260929 18:38:23.859285 15676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 18:38:23.860313 15676 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe855e489ad google::LogMessage::Fail() at ??:0
@ 0x7fe855e4b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe855e484de google::LogMessage::Flush() at ??:0
@ 0x7fe855e49a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe860787fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe860788d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe860785579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe8607c9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe85809a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe85809ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe85809c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe85808b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe8596cf6db start_thread at ??:0
@ 0x7fe85494661f clone at ??:0
W20260929 18:38:23.944895 15669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:23.945253 15669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:23.945325 15669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:23.951412 15669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:23.951499 15669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:23.951594 15669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:38:23.959570 15669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:41225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=37119
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:23.961071 15669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:23.962476 15669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:23.974014 15669 server_base.cc:1034] running on GCE node
W20260929 18:38:23.973899 15708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:23.974160 15711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:23.973899 15709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:23.974699 15669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:23.975349 15669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:23.976554 15669 hybrid_clock.cc:648] HybridClock initialized: now 1790707103976530 us; error 44 us; skew 500 ppm
Sep 29 18:38:23 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:23.979349 15669 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:38:23.980239 15669 webserver.cc:460] Webserver started at http://127.0.198.1:37119/ using document root <none> and password file <none>
I20260929 18:38:23.980772 15669 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:23.980834 15669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:23.984903 15669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 18:38:23.987291 15718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:23.988754 15669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:38:23.988873 15669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "456b28faff91441598be5a2d79886ef6"
format_stamp: "Formatted at 2026-09-29 18:38:21 on dist-test-slave-q5qx"
server_key: "f612f50ebe1ebbf0c6364a484a39173c"
server_key_iv: "3c2a251abee3b65f0e704b68fa99c075"
server_key_version: "encryptionkey@0"
I20260929 18:38:23.989421 15669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:24.010816 15669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:24.014060 15669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:24.014243 15669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:24.014869 15669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:24.016017 15725 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:38:24.021611 15669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:38:24.021699 15669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 18:38:24.021773 15669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:38:24.023589 15669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:38:24.023644 15669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:38:24.023970 15725 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Bootstrap starting.
I20260929 18:38:24.033829 15669 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:41225
I20260929 18:38:24.033933 15832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:41225 every 8 connection(s)
I20260929 18:38:24.035226 15669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:24.035393 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15669
I20260929 18:38:24.035815 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:35537
--local_ip_for_outbound_sockets=127.0.198.2
--tserver_master_addrs=127.0.198.62:41069
--webserver_port=40989
--webserver_interface=127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:24.043926 15725 log.cc:824] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:24.050980 15725 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:38:24.051307 15725 tablet_bootstrap.cc:492] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Bootstrap complete.
I20260929 18:38:24.052213 15725 ts_tablet_manager.cc:1397] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.009s
I20260929 18:38:24.054644 15725 raft_consensus.cc:348] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:24.055056 15725 raft_consensus.cc:729] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456b28faff91441598be5a2d79886ef6, State: Initialized, Role: FOLLOWER
I20260929 18:38:24.055539 15725 consensus_queue.cc:260] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:24.056470 15725 ts_tablet_manager.cc:1428] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260929 18:38:24.126408 15836 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:24.038422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:55823)
0929 18:38:24.039082 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.039150 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:24.040287 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:24.125377 (+ 85090us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":160,"threads_started":1}
W20260929 18:38:24.126962 15833 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:41069 (0 consecutive failures): Network error: Failed to ping master at 127.0.198.62:41069: Client connection negotiation failed: client connection to 127.0.198.62:41069: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:38:24.127701 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.127430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:33155)
0929 18:38:24.127561 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.127637 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 18:38:24.128403 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.128165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:58117)
0929 18:38:24.128296 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.128346 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:38:24.128628 15833 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:38:24.177130 15835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:24.177415 15835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:24.177459 15835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:24.181582 15835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:24.181655 15835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:24.181715 15835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:38:24.187027 15835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:35537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=40989
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:24.188067 15835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:24.189044 15835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:24.198580 15835 server_base.cc:1034] running on GCE node
W20260929 18:38:24.198570 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:24.198511 15844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:24.198511 15846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:24.199263 15835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:24.199909 15835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:24.201081 15835 hybrid_clock.cc:648] HybridClock initialized: now 1790707104201060 us; error 35 us; skew 500 ppm
Sep 29 18:38:24 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:24.203884 15835 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:38:24.204753 15835 webserver.cc:460] Webserver started at http://127.0.198.2:40989/ using document root <none> and password file <none>
I20260929 18:38:24.205276 15835 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:24.205338 15835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:24.209195 15835 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:24.211452 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:24.212723 15835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:38:24.212826 15835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "c7a40c8f558a468e87fe0e6438b8d424"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "0348692a8ac11d0515af282f26449ad5"
server_key_iv: "9edcd1b9930eeae594939185cd2bc1dc"
server_key_version: "encryptionkey@0"
I20260929 18:38:24.213481 15835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:24.229841 15835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:24.232977 15835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:24.233162 15835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:24.233752 15835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:24.234599 15835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:24.234647 15835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:24.234702 15835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:24.234735 15835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:24.243664 15835 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:35537
I20260929 18:38:24.243690 15966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:35537 every 8 connection(s)
I20260929 18:38:24.244669 15835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:24.248535 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.246850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:36023)
0929 18:38:24.247310 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.247799 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
W20260929 18:38:24.248991 15967 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:41069 (0 consecutive failures): Network error: Failed to ping master at 127.0.198.62:41069: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
I20260929 18:38:24.249577 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.249377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:59941)
0929 18:38:24.249479 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.249524 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:38:24.250130 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.249932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:55025)
0929 18:38:24.250042 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.250084 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:38:24.250267 15967 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:38:24.254504 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15835
I20260929 18:38:24.254824 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:39505
--local_ip_for_outbound_sockets=127.0.198.3
--tserver_master_addrs=127.0.198.62:41069
--webserver_port=41863
--webserver_interface=127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:24.364528 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:24.364809 15970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:24.364851 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:24.368835 15970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:24.368901 15970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:24.368959 15970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:38:24.374289 15970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:39505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=41863
--tserver_master_addrs=127.0.198.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:24.375313 15970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:24.376325 15970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:24.386143 15970 server_base.cc:1034] running on GCE node
W20260929 18:38:24.386088 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:24.386088 15975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:24.386109 15976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:24.386824 15970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:24.387521 15970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:24.388701 15970 hybrid_clock.cc:648] HybridClock initialized: now 1790707104388672 us; error 44 us; skew 500 ppm
Sep 29 18:38:24 dist-test-slave-q5qx krb5kdc[14524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:24.391557 15970 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:38:24.392494 15970 webserver.cc:460] Webserver started at http://127.0.198.3:41863/ using document root <none> and password file <none>
I20260929 18:38:24.393028 15970 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:24.393091 15970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:24.396961 15970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:38:24.398981 15985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:24.400193 15970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:24.400298 15970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "c3a43de10c0b45aebf1284e1f1b41a40"
format_stamp: "Formatted at 2026-09-29 18:38:22 on dist-test-slave-q5qx"
server_key: "c20f0c281847c500ff9066f1cab0bc7b"
server_key_iv: "c4db0e6e25fb98dea892815b6c24c77c"
server_key_version: "encryptionkey@0"
I20260929 18:38:24.400820 15970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:24.422360 15970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:24.425424 15970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:24.425585 15970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:24.426158 15970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:24.426939 15970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:24.426983 15970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:24.427021 15970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:24.427040 15970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:24.435544 15970 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:39505
I20260929 18:38:24.435581 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:39505 every 8 connection(s)
I20260929 18:38:24.436532 15970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:24.440558 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.438753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:43827)
0929 18:38:24.439241 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.439375 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":114,"threads_started":1}
W20260929 18:38:24.441144 16099 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:41069 (0 consecutive failures): Network error: Failed to ping master at 127.0.198.62:41069: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
I20260929 18:38:24.441232 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 15970
I20260929 18:38:24.441800 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.441575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:33267)
0929 18:38:24.441693 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.441744 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:38:24.442462 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:24.442287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:43253)
0929 18:38:24.442375 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:24.442413 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 18:38:24.442615 16099 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:38:25.129671 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:25.129232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:59391)
0929 18:38:25.129488 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:25.129557 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260929 18:38:25.251257 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:25.250839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:33335)
0929 18:38:25.251090 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:25.251153 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260929 18:38:25.371604 16105 raft_consensus.cc:482] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:25.371791 16105 raft_consensus.cc:504] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:25.372782 16105 leader_election.cc:288] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505), c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537)
Sep 29 18:38:25 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.3@KRBTEST.COM
Sep 29 18:38:25 dist-test-slave-q5qx krb5kdc[14524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.2@KRBTEST.COM
I20260929 18:38:25.386037 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:25.373056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39505 (local address 127.0.198.1:46731)
0929 18:38:25.373239 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:25.373255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:25.373378 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:25.375210 (+ 1832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:25.375216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:25.375738 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:25.376344 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:25.376375 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.377585 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.377587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:25.378546 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:25.378555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.379200 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.379204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:25.379387 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:25.379969 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:25.379990 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:25.381529 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:25.383792 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:25.383799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:25.383813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:25.384047 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:25.384369 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:25.384372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:25.384374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:25.384500 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:25.385104 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:25.385110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:25.385328 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 18:38:25.385491 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 18:38:25.385823 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":124,"spinlock_wait_cycles":35712}
I20260929 18:38:25.386057 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:25.373256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:46731 (local address 127.0.198.3:39505)
0929 18:38:25.373691 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:25.373707 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:25.373755 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:38:25.374471 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:25.374478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:25.374853 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:25.375095 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:25.376509 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.377462 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.378655 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.379091 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.379231 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:25.381753 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:25.381758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:25.381770 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:25.381796 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:25.383679 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:25.384151 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:25.384157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:25.384162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:25.384275 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:25.384583 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:25.384587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:25.384588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:25.384995 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:25.385111 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:25.385615 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 18:38:25.385857 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":107,"threads_started":1}
I20260929 18:38:25.386087 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:25.373314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:35537 (local address 127.0.198.1:41297)
0929 18:38:25.373595 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:25.373605 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:25.373676 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:25.375210 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:25.375216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:25.375738 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:25.376345 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:25.376368 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.377589 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.377592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:25.378541 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:25.378550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.379203 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.379207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:25.379384 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:25.380083 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:25.380094 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:25.381885 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:25.384055 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:25.384058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:25.384060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:25.384241 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:25.384552 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:25.384555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:25.384557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:25.384630 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:25.385171 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:25.385174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:25.385329 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 18:38:25.385491 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 18:38:25.385916 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":273,"spinlock_wait_cycles":14080,"thread_start_us":75,"threads_started":1}
I20260929 18:38:25.386065 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:25.373615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:41297 (local address 127.0.198.2:35537)
0929 18:38:25.373890 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:25.373912 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:25.373942 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:38:25.374634 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:25.374641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:25.374948 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:25.375213 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:25.376512 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.377449 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.378650 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:25.379094 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:25.379232 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:25.382065 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:25.382069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:25.382080 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:25.382107 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:25.383934 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:25.384348 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:25.384353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:25.384357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:25.384462 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:25.384707 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:25.384711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:25.384712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:25.385068 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:25.385178 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:25.385627 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 18:38:25.385858 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
W20260929 18:38:25.389245 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
W20260929 18:38:25.392786 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:25.392921 15722 leader_election.cc:302] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 456b28faff91441598be5a2d79886ef6; no voters: c3a43de10c0b45aebf1284e1f1b41a40, c7a40c8f558a468e87fe0e6438b8d424
I20260929 18:38:25.393186 16105 raft_consensus.cc:2726] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:38:25.443542 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:25.443121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:34773)
0929 18:38:25.443379 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:25.443439 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260929 18:38:26.130924 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:26.130443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:50709)
0929 18:38:26.130723 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:26.130789 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260929 18:38:26.252590 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:26.252099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:38783)
0929 18:38:26.252429 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:26.252500 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260929 18:38:26.444752 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:26.444288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:60257)
0929 18:38:26.444582 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:26.444643 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260929 18:38:27.132166 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:27.131716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:44663)
0929 18:38:27.132002 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:27.132068 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260929 18:38:27.253786 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:27.253333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:38861)
0929 18:38:27.253611 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:27.253672 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":71,"threads_started":1}
I20260929 18:38:27.446071 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:27.445582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:43295)
0929 18:38:27.445860 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:27.445933 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20260929 18:38:27.606130 16116 raft_consensus.cc:482] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:27.606261 16116 raft_consensus.cc:504] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:27.606684 16116 leader_election.cc:288] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505), c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537)
W20260929 18:38:27.607420 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
W20260929 18:38:27.607542 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:27.607591 15722 leader_election.cc:302] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 456b28faff91441598be5a2d79886ef6; no voters: c3a43de10c0b45aebf1284e1f1b41a40, c7a40c8f558a468e87fe0e6438b8d424
I20260929 18:38:27.607749 16116 raft_consensus.cc:2726] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:38:28.133441 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:28.132965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:41371)
0929 18:38:28.133252 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:28.133323 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260929 18:38:28.255038 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:28.254630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:34097)
0929 18:38:28.254885 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:28.254943 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260929 18:38:28.447335 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:28.446895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:37565)
0929 18:38:28.447176 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:28.447239 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20260929 18:38:29.135175 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:29.134583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:53927)
0929 18:38:29.134973 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:29.135043 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":79,"threads_started":1}
I20260929 18:38:29.256187 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:29.255786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:50829)
0929 18:38:29.256033 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:29.256090 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260929 18:38:29.448755 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:29.448284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:43271)
0929 18:38:29.448593 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:29.448656 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260929 18:38:30.123998 16123 raft_consensus.cc:482] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:30.124143 16123 raft_consensus.cc:504] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:30.124539 16123 leader_election.cc:288] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505), c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537)
W20260929 18:38:30.125356 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
W20260929 18:38:30.125504 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:30.125546 15722 leader_election.cc:302] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 456b28faff91441598be5a2d79886ef6; no voters: c3a43de10c0b45aebf1284e1f1b41a40, c7a40c8f558a468e87fe0e6438b8d424
I20260929 18:38:30.125705 16123 raft_consensus.cc:2726] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:38:30.136369 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:30.135913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:56695)
0929 18:38:30.136211 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:30.136274 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":142,"threads_started":1}
I20260929 18:38:30.257577 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:30.257122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:56351)
0929 18:38:30.257409 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:30.257472 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":77,"threads_started":1}
I20260929 18:38:30.450205 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:30.449641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:53425)
0929 18:38:30.450036 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:30.450106 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":80,"threads_started":1}
I20260929 18:38:31.137646 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:31.137168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:52509)
0929 18:38:31.137476 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:31.137549 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260929 18:38:31.258858 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:31.258376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:54979)
0929 18:38:31.258698 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:31.258760 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":93,"threads_started":1}
I20260929 18:38:31.451381 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:31.450934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:41141)
0929 18:38:31.451230 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:31.451286 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":72,"threads_started":1}
I20260929 18:38:32.138913 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:32.138437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:35919)
0929 18:38:32.138732 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:32.138816 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260929 18:38:32.167637 16131 raft_consensus.cc:482] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:32.167766 16131 raft_consensus.cc:504] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:32.168171 16131 leader_election.cc:288] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505), c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537)
W20260929 18:38:32.168998 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
W20260929 18:38:32.169128 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:32.169189 15722 leader_election.cc:302] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 456b28faff91441598be5a2d79886ef6; no voters: c3a43de10c0b45aebf1284e1f1b41a40, c7a40c8f558a468e87fe0e6438b8d424
I20260929 18:38:32.169337 16131 raft_consensus.cc:2726] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:38:32.260221 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:32.259727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:46311)
0929 18:38:32.260045 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:32.260119 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
I20260929 18:38:32.452623 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:32.452179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:32795)
0929 18:38:32.452466 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:32.452530 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
I20260929 18:38:33.140377 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:33.139762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:37457)
0929 18:38:33.140072 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:33.140148 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260929 18:38:33.261587 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:33.261093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:43755)
0929 18:38:33.261419 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:33.261490 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
I20260929 18:38:33.454090 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:33.453511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:57583)
0929 18:38:33.453869 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:33.453938 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
I20260929 18:38:34.141583 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:34.141196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:57115)
0929 18:38:34.141427 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:34.141487 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260929 18:38:34.262900 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:34.262489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:54229)
0929 18:38:34.262745 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:34.262804 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260929 18:38:34.455361 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:34.454902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:39627)
0929 18:38:34.455191 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:34.455254 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260929 18:38:35.142755 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:35.142372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:44967)
0929 18:38:35.142606 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:35.142659 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":123,"threads_started":1}
I20260929 18:38:35.264084 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:35.263687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:39659)
0929 18:38:35.263928 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:35.263988 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20260929 18:38:35.456612 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:35.456190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:59817)
0929 18:38:35.456451 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:35.456514 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260929 18:38:36.144001 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:36.143572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:44009)
0929 18:38:36.143825 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:36.143882 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260929 18:38:36.265341 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:36.264909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:37719)
0929 18:38:36.265186 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:36.265246 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
I20260929 18:38:36.457962 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:36.457504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:38181)
0929 18:38:36.457781 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:36.457847 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260929 18:38:37.145103 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:37.144706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:36529)
0929 18:38:37.144955 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:37.145009 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260929 18:38:37.266607 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:37.266140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:41155)
0929 18:38:37.266381 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:37.266507 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260929 18:38:37.459246 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:37.458779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:55331)
0929 18:38:37.459066 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:37.459146 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260929 18:38:38.146345 16149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:38.145876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:43535)
0929 18:38:38.146191 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:38.146252 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260929 18:38:38.267858 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:38.267385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:35517)
0929 18:38:38.267689 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:38.267753 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260929 18:38:38.407780 16151 raft_consensus.cc:482] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:38.407912 16151 raft_consensus.cc:504] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a43de10c0b45aebf1284e1f1b41a40" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39505 } } peers { permanent_uuid: "c7a40c8f558a468e87fe0e6438b8d424" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 35537 } } peers { permanent_uuid: "456b28faff91441598be5a2d79886ef6" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 41225 } }
I20260929 18:38:38.408327 16151 leader_election.cc:288] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505), c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537)
W20260929 18:38:38.409138 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a43de10c0b45aebf1284e1f1b41a40 (127.0.198.3:39505): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
W20260929 18:38:38.409271 15722 leader_election.cc:341] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7a40c8f558a468e87fe0e6438b8d424 (127.0.198.2:35537): Not found: Tablet not found: ae1ae1980ffb4a329bd9d143a8802508
I20260929 18:38:38.409322 15722 leader_election.cc:302] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 456b28faff91441598be5a2d79886ef6; no voters: c3a43de10c0b45aebf1284e1f1b41a40, c7a40c8f558a468e87fe0e6438b8d424
I20260929 18:38:38.409507 16151 raft_consensus.cc:2726] T ae1ae1980ffb4a329bd9d143a8802508 P 456b28faff91441598be5a2d79886ef6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:38:38.460551 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:38.460087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.3:45981)
0929 18:38:38.460347 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:38.460407 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260929 18:38:39.147778 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:39.147311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.1:36421)
0929 18:38:39.147614 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:39.147676 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260929 18:38:39.269260 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:39.268838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41069 (local address 127.0.198.2:48305)
0929 18:38:39.269107 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:39.269168 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
I20260929 18:38:39.449064 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 15669
I20260929 18:38:39.453742 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 15835
I20260929 18:38:39.458424 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 15970
2026-09-29T18:38:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19103 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:38:39 dist-test-slave-q5qx krb5kdc[16157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:38:39 dist-test-slave-q5qx krb5kdc[16157](info): set up 2 sockets
Sep 29 18:38:39 dist-test-slave-q5qx krb5kdc[16157](info): commencing operation
krb5kdc: starting...
W20260929 18:38:40.426149 792 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:38:40 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:38:40.544843 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38905
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:38905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:40.652032 16170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:40.652309 16170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:40.652366 16170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:40.656374 16170 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:38:40.656427 16170 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:38:40.656447 16170 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:38:40.656466 16170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:40.656486 16170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:40.656512 16170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:40.656541 16170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:40.656569 16170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:40.661800 16170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:38905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:40.662909 16170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:40.663964 16170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:40.672537 16175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:40.672601 16178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:40.672612 16176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:40.672684 16170 server_base.cc:1034] running on GCE node
I20260929 18:38:40.673382 16170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:40.674619 16170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:40.675786 16170 hybrid_clock.cc:648] HybridClock initialized: now 1790707120675769 us; error 35 us; skew 500 ppm
Sep 29 18:38:40 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:40.678488 16170 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:38:40.679353 16170 webserver.cc:460] Webserver started at http://127.0.198.62:38901/ using document root <none> and password file <none>
I20260929 18:38:40.680130 16170 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:40.680226 16170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:40.680486 16170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:40.682616 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "43356f430fac4f4a85000ce1ec490e92"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "e28a2070f42930925c92955383da20ef"
server_key_iv: "d3c9c76a3a057543929fe09c5a1adf57"
server_key_version: "encryptionkey@0"
I20260929 18:38:40.683164 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "43356f430fac4f4a85000ce1ec490e92"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "e28a2070f42930925c92955383da20ef"
server_key_iv: "d3c9c76a3a057543929fe09c5a1adf57"
server_key_version: "encryptionkey@0"
I20260929 18:38:40.686956 16170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:38:40.689332 16185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:40.690649 16170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:40.690752 16170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "43356f430fac4f4a85000ce1ec490e92"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "e28a2070f42930925c92955383da20ef"
server_key_iv: "d3c9c76a3a057543929fe09c5a1adf57"
server_key_version: "encryptionkey@0"
I20260929 18:38:40.690852 16170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:40.705181 16170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:40.708331 16170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:40.708565 16170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:40.715323 16170 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:38905
I20260929 18:38:40.715631 16237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:38905 every 8 connection(s)
I20260929 18:38:40.716338 16170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:40.719168 16238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:40.720988 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 16170
I20260929 18:38:40.721094 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:38:40.721462 792 external_mini_cluster.cc:1428] Setting key c8a00a5ade031ab876b8bf79a9f00ac5
I20260929 18:38:40.726231 16238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Bootstrap starting.
I20260929 18:38:40.728844 16238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:40.729712 16238 log.cc:824] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:38:40 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:40.731809 16238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: No bootstrap required, opened a new log
I20260929 18:38:40.736485 16238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:38:40.736786 16238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:40.736842 16238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43356f430fac4f4a85000ce1ec490e92, State: Initialized, Role: FOLLOWER
I20260929 18:38:40.737006 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:40.722971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.0.198.62:38905)
0929 18:38:40.723414 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:40.723438 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:40.723482 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:38:40.724264 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:40.724288 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:40.724728 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:40.725122 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:40.725806 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:40.727285 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:40.728176 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:40.728861 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:40.729021 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:40.731696 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:40.731701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:40.731718 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:40.731748 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:40.733665 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:40.734117 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:40.734123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:40.734129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:40.734288 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:40.734594 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:40.734597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:40.734598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:40.735115 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:40.735315 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:40.735590 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:38:40.735920 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260929 18:38:40.737401 16238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:38:40.737540 16238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:40.737586 16238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:40.737658 16238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:40.738574 16238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:38:40.738898 16238 leader_election.cc:302] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43356f430fac4f4a85000ce1ec490e92; no voters:
I20260929 18:38:40.739187 16238 leader_election.cc:288] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:40.739276 16243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:40.739439 16243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 LEADER]: Becoming Leader. State: Replica: 43356f430fac4f4a85000ce1ec490e92, State: Running, Role: LEADER
I20260929 18:38:40.739737 16243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:38:40.740449 16238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:40.741358 16245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43356f430fac4f4a85000ce1ec490e92. Latest consensus state: current_term: 1 leader_uuid: "43356f430fac4f4a85000ce1ec490e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } } }
I20260929 18:38:40.741498 16245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:40.741918 16252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:40.742431 16244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43356f430fac4f4a85000ce1ec490e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } } }
I20260929 18:38:40.742519 16244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:40.745086 16252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:40.750917 16252 catalog_manager.cc:1347] Generated new cluster ID: 496e6c74362241cda8a84b9697e8e3d3
I20260929 18:38:40.751001 16252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:40.761071 16252 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:40.762388 16252 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:40.771559 16252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Generated new TSK 0
I20260929 18:38:40.772253 16252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:38:40.805451 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38905
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:40.950902 16264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:40.951174 16264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:40.951216 16264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:40.955206 16264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:40.955269 16264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:40.955330 16264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:38:40.960572 16264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:40.961601 16264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:40.962591 16264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:40.971858 16270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:40.971880 16269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:40.971899 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:40.972162 16264 server_base.cc:1034] running on GCE node
I20260929 18:38:40.972646 16264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:40.973277 16264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:40.974476 16264 hybrid_clock.cc:648] HybridClock initialized: now 1790707120974457 us; error 33 us; skew 500 ppm
Sep 29 18:38:40 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:40.977283 16264 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:38:40.978287 16264 webserver.cc:460] Webserver started at http://127.0.198.1:41493/ using document root <none> and password file <none>
I20260929 18:38:40.978829 16264 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:40.978894 16264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:40.979096 16264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:40.981087 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "2da3c2d0cdea4bb6b16db539dc5a4905"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "797f96685fed95abce93425a1898b419"
server_key_iv: "636cbf56aa33cf963926ac33319135ac"
server_key_version: "encryptionkey@0"
I20260929 18:38:40.981626 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "2da3c2d0cdea4bb6b16db539dc5a4905"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "797f96685fed95abce93425a1898b419"
server_key_iv: "636cbf56aa33cf963926ac33319135ac"
server_key_version: "encryptionkey@0"
I20260929 18:38:40.985234 16264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:38:40.987753 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:40.989022 16264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:40.989149 16264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "2da3c2d0cdea4bb6b16db539dc5a4905"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "797f96685fed95abce93425a1898b419"
server_key_iv: "636cbf56aa33cf963926ac33319135ac"
server_key_version: "encryptionkey@0"
I20260929 18:38:40.989289 16264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:41.023434 16264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:41.026978 16264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:41.027173 16264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:41.027776 16264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:41.028616 16264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:41.028663 16264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.028716 16264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:41.028743 16264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.037475 16264 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:35053
I20260929 18:38:41.037521 16392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:35053 every 8 connection(s)
I20260929 18:38:41.038399 16264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:41.042943 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 16264
I20260929 18:38:41.043064 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:38:41.043438 792 external_mini_cluster.cc:1428] Setting key 5355bc4275c7bf81e4b9687032b29e33
Sep 29 18:38:41 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:38:41.052706 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.040142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:51511 (local address 127.0.198.62:38905)
0929 18:38:41.040294 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.040298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.041168 (+ 870us) server_negotiation.cc:407] Connection header received
0929 18:38:41.042549 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.042552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.042607 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.042681 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:41.044012 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.044706 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.045862 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.046222 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.046328 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.049074 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:41.049077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:41.049079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:41.049104 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:41.050636 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.051165 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.051168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.051176 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.051250 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.051564 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.051568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.051569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.051743 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:41.051827 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.052394 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.052537 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:38:41.053527 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.040533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:51511)
0929 18:38:41.040990 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.041034 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.042332 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.042877 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.042883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.043285 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:41.043806 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.043827 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.044837 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.044841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.045737 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.045747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.046407 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.046411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.046615 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.047265 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:41.047286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:41.048904 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:41.050765 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:41.050771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:41.050788 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:41.051068 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:41.051332 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:41.051335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:41.051337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:41.051476 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:41.051841 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:41.051848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:41.052153 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.052331 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.052663 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:38:41.054867 16393 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38905
I20260929 18:38:41.055109 16393 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:41.055567 16393 heartbeater.cc:511] Master 127.0.198.62:38905 requested a full tablet report, sending...
I20260929 18:38:41.057195 16202 ts_manager.cc:194] Registered new tserver with Master: 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053)
I20260929 18:38:41.058736 16202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:51511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:38:41.070729 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38905
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:41.180372 16398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:41.180631 16398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:41.180672 16398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:41.184690 16398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:41.184749 16398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:41.184808 16398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:38:41.190033 16398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:41.191071 16398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:41.192162 16398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:41.201800 16406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:41.201763 16404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:41.201773 16403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:41.201790 16398 server_base.cc:1034] running on GCE node
I20260929 18:38:41.202401 16398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:41.203051 16398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:41.204216 16398 hybrid_clock.cc:648] HybridClock initialized: now 1790707121204195 us; error 31 us; skew 500 ppm
Sep 29 18:38:41 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:41.206908 16398 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:38:41.207834 16398 webserver.cc:460] Webserver started at http://127.0.198.2:34983/ using document root <none> and password file <none>
I20260929 18:38:41.208387 16398 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:41.208451 16398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:41.208635 16398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:41.210808 16398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "5b7738d7ac7d4ad286863268d3f0ff25"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "926f4e859363a2d974510f161ef5ed11"
server_key_iv: "2bfa37bc1c23c4280f644a1a1ff146a1"
server_key_version: "encryptionkey@0"
I20260929 18:38:41.211374 16398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "5b7738d7ac7d4ad286863268d3f0ff25"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "926f4e859363a2d974510f161ef5ed11"
server_key_iv: "2bfa37bc1c23c4280f644a1a1ff146a1"
server_key_version: "encryptionkey@0"
I20260929 18:38:41.215111 16398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:38:41.217489 16413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.218703 16398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:41.218808 16398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "5b7738d7ac7d4ad286863268d3f0ff25"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "926f4e859363a2d974510f161ef5ed11"
server_key_iv: "2bfa37bc1c23c4280f644a1a1ff146a1"
server_key_version: "encryptionkey@0"
I20260929 18:38:41.218900 16398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:41.240442 16398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:41.243759 16398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:41.243945 16398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:41.244524 16398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:41.245328 16398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:41.245374 16398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.245414 16398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:41.245427 16398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.253631 16398 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:38317
I20260929 18:38:41.253657 16526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:38317 every 8 connection(s)
I20260929 18:38:41.254585 16398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:41.256949 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 16398
I20260929 18:38:41.257056 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:38:41.257359 792 external_mini_cluster.cc:1428] Setting key b84564afb94988f35e7b253c34dfc73b
Sep 29 18:38:41 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:38:41.269654 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.256532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:35865 (local address 127.0.198.62:38905)
0929 18:38:41.256675 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.256679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.257558 (+ 879us) server_negotiation.cc:407] Connection header received
0929 18:38:41.259030 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.259033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.259076 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.259145 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:41.260681 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.261476 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.262676 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.262947 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.263049 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.265996 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:41.266001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:41.266004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:41.266032 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:41.267452 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.267996 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.268000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.268002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.268078 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.268494 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.268497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.268498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.268694 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:41.268781 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.269347 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.269487 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:38:41.270558 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.256881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:35865)
0929 18:38:41.257379 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.257442 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.258823 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.259337 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.259346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.259834 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:41.260503 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.260521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.261609 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.261612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.262560 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.262569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.263121 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.263126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.263406 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.264085 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:41.264102 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:41.265784 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:41.267583 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:41.267589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:41.267606 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:41.267901 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:41.268159 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:41.268163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:41.268165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:41.268334 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:41.268797 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:41.268804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:41.269108 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.269289 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.269638 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:38:41.271831 16527 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38905
I20260929 18:38:41.272078 16527 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:41.272581 16527 heartbeater.cc:511] Master 127.0.198.62:38905 requested a full tablet report, sending...
I20260929 18:38:41.273667 16202 ts_manager.cc:194] Registered new tserver with Master: 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317)
I20260929 18:38:41.274420 16202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:35865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:38:41.286789 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38905
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:41.396592 16532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:41.396862 16532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:41.396903 16532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:41.400930 16532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:41.400998 16532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:41.401055 16532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:38:41.406482 16532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:38:41.407516 16532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:41.408591 16532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:41.418165 16538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:41.418212 16532 server_base.cc:1034] running on GCE node
W20260929 18:38:41.418179 16537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:41.418179 16540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:41.418859 16532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:41.419488 16532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:41.420647 16532 hybrid_clock.cc:648] HybridClock initialized: now 1790707121420623 us; error 38 us; skew 500 ppm
Sep 29 18:38:41 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:41.423437 16532 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:38:41.424336 16532 webserver.cc:460] Webserver started at http://127.0.198.3:44933/ using document root <none> and password file <none>
I20260929 18:38:41.424839 16532 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:41.424892 16532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:41.425060 16532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:41.427102 16532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "a750095c0ccd49de8ef4f2d7dc8901ba"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "e977599c18c3ec73368ec4cf3fe5ee37"
server_key_iv: "b0060a220de6f22c3702940ac91582d4"
server_key_version: "encryptionkey@0"
I20260929 18:38:41.427615 16532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "a750095c0ccd49de8ef4f2d7dc8901ba"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "e977599c18c3ec73368ec4cf3fe5ee37"
server_key_iv: "b0060a220de6f22c3702940ac91582d4"
server_key_version: "encryptionkey@0"
I20260929 18:38:41.431278 16532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:38:41.433612 16547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.434839 16532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:41.434933 16532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "a750095c0ccd49de8ef4f2d7dc8901ba"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "e977599c18c3ec73368ec4cf3fe5ee37"
server_key_iv: "b0060a220de6f22c3702940ac91582d4"
server_key_version: "encryptionkey@0"
I20260929 18:38:41.435039 16532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:41.449080 16532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:41.452198 16532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:41.452349 16532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:41.452919 16532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:41.453707 16532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:41.453751 16532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.453788 16532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:41.453804 16532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:41.462307 16532 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:33715
I20260929 18:38:41.462334 16660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:33715 every 8 connection(s)
I20260929 18:38:41.463238 16532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:38:41 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:38:41.472954 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 16532
I20260929 18:38:41.473061 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:38:41.473328 792 external_mini_cluster.cc:1428] Setting key c35d73b632e9c6591ca4eee515cfc41d
I20260929 18:38:41.476953 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.465048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:53279 (local address 127.0.198.62:38905)
0929 18:38:41.465185 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.465189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.466094 (+ 905us) server_negotiation.cc:407] Connection header received
0929 18:38:41.467395 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.467399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.467458 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.467535 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:41.468746 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.469274 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.470494 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.470717 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.470804 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.473374 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:41.473378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:41.473381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:41.473412 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:41.474937 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.475514 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.475517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.475519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.475598 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.475917 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.475920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.475922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.476105 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:41.476204 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.476679 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.476796 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:38:41.477744 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.465419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:53279)
0929 18:38:41.465916 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.465961 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.467165 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.467729 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.467737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.468110 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:41.468594 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.468608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.469397 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.469400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.470373 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.470382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.470861 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.470865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.471024 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.471630 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:41.471646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:41.473194 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:41.475053 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:41.475060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:41.475076 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:41.475420 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:41.475679 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:41.475682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:41.475684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:41.475826 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:41.476202 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:41.476214 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:41.476460 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.476626 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.476962 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260929 18:38:41.478868 16661 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38905
I20260929 18:38:41.479105 16661 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:41.479599 16661 heartbeater.cc:511] Master 127.0.198.62:38905 requested a full tablet report, sending...
I20260929 18:38:41.480640 16202 ts_manager.cc:194] Registered new tserver with Master: a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715)
I20260929 18:38:41.481187 16202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:53279
I20260929 18:38:41.487238 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:38:41.496328 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.489062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.0.198.62:38905)
0929 18:38:41.489202 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.489206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.489312 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:38:41.489417 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.489420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.489458 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.489522 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:41.490201 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.490730 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.491556 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.491779 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.491864 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.493225 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:41.493229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:41.493232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:41.493268 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:41.494701 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.495096 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.495100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.495102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.495176 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:41.495460 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:41.495463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:41.495465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:41.495743 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:41.495866 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.496073 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.496178 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:38:41.499131 16202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:38:41.502314 16202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:38:41.516549 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.511575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.0.198.3:33715)
0929 18:38:41.511939 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.511951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.511974 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:38:41.512050 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.512055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.512253 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.512813 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:41.512954 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.513685 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.515176 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.515902 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.515997 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.516145 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.516315 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.516407 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":164,"threads_started":1}
I20260929 18:38:41.516544 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.511086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:33715 (local address 127.0.0.1:43324)
0929 18:38:41.511766 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.511777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.511844 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.512408 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.512410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.512440 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:41.512631 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.512639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.513802 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.513805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.515044 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.515054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.516008 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.516012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.516128 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.516170 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.516267 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.516394 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":22,"spinlock_wait_cycles":6784,"thread_start_us":80,"threads_started":1}
I20260929 18:38:41.516808 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.510890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:38317 (local address 127.0.0.1:44250)
0929 18:38:41.511618 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.511631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.511706 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.512719 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.512722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.512738 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:41.512874 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.512880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.514334 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.514336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.515339 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.515374 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.516444 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.516447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.516533 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.516556 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.516603 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.516671 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":41,"thread_start_us":101,"threads_started":1}
I20260929 18:38:41.516996 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.510758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:35053 (local address 127.0.0.1:33384)
0929 18:38:41.511409 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.511460 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.511587 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.512396 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.512402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.512438 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:41.512885 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.512892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.514320 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.514323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.515515 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.515524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.516651 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.516655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.516742 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.516762 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.516823 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.516885 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":355,"spinlock_wait_cycles":4224,"thread_start_us":81,"threads_started":1}
I20260929 18:38:41.517418 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.512001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.0.198.2:38317)
0929 18:38:41.512320 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.512332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.512351 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:38:41.512416 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.512420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.512593 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.512725 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:41.512980 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.514178 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.515506 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.516307 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.516996 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.517142 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.517193 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.517284 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
I20260929 18:38:41.517464 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.510933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.198.1:35053)
0929 18:38:41.511942 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.511962 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.511980 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:38:41.512042 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.512046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.512228 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.512542 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:41.513015 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.514175 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.515628 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.516365 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.516463 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.516599 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.517248 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.517332 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":88,"threads_started":1}
I20260929 18:38:41.519944 16461 tablet_service.cc:1467] Processing CreateTablet for tablet fcc0d148124f466ca26792d4079a438d (DEFAULT_TABLE table=test-table [id=2aa4888c10a54e5caabef3e75a2357e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:41.519979 16595 tablet_service.cc:1467] Processing CreateTablet for tablet fcc0d148124f466ca26792d4079a438d (DEFAULT_TABLE table=test-table [id=2aa4888c10a54e5caabef3e75a2357e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:41.519982 16327 tablet_service.cc:1467] Processing CreateTablet for tablet fcc0d148124f466ca26792d4079a438d (DEFAULT_TABLE table=test-table [id=2aa4888c10a54e5caabef3e75a2357e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:41.520882 16461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc0d148124f466ca26792d4079a438d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:41.520952 16595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc0d148124f466ca26792d4079a438d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:41.520956 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc0d148124f466ca26792d4079a438d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:41.527531 16678 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Bootstrap starting.
I20260929 18:38:41.528482 16680 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Bootstrap starting.
I20260929 18:38:41.529778 16678 tablet_bootstrap.cc:654] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:41.530309 16679 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap starting.
I20260929 18:38:41.530516 16680 tablet_bootstrap.cc:654] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:41.530543 16678 log.cc:824] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:41.531306 16680 log.cc:824] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:41.532248 16679 tablet_bootstrap.cc:654] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:41.532406 16678 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: No bootstrap required, opened a new log
I20260929 18:38:41.532582 16678 ts_tablet_manager.cc:1397] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 18:38:41.533234 16680 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: No bootstrap required, opened a new log
I20260929 18:38:41.533354 16679 log.cc:824] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:41.533412 16680 ts_tablet_manager.cc:1397] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:38:41.535286 16679 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: No bootstrap required, opened a new log
I20260929 18:38:41.535467 16679 ts_tablet_manager.cc:1397] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:38:41.535656 16678 raft_consensus.cc:348] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.535887 16678 raft_consensus.cc:374] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:41.535938 16678 raft_consensus.cc:729] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7738d7ac7d4ad286863268d3f0ff25, State: Initialized, Role: FOLLOWER
I20260929 18:38:41.536322 16678 consensus_queue.cc:260] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.536898 16527 heartbeater.cc:503] Master 127.0.198.62:38905 was elected leader, sending a full tablet report...
I20260929 18:38:41.536957 16680 raft_consensus.cc:348] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.537197 16680 raft_consensus.cc:374] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:41.537241 16680 raft_consensus.cc:729] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a750095c0ccd49de8ef4f2d7dc8901ba, State: Initialized, Role: FOLLOWER
I20260929 18:38:41.537250 16678 ts_tablet_manager.cc:1428] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 18:38:41.537724 16680 consensus_queue.cc:260] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.538381 16661 heartbeater.cc:503] Master 127.0.198.62:38905 was elected leader, sending a full tablet report...
I20260929 18:38:41.538734 16680 ts_tablet_manager.cc:1428] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260929 18:38:41.539036 16679 raft_consensus.cc:348] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.539258 16679 raft_consensus.cc:374] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:41.539309 16679 raft_consensus.cc:729] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da3c2d0cdea4bb6b16db539dc5a4905, State: Initialized, Role: FOLLOWER
I20260929 18:38:41.539721 16679 consensus_queue.cc:260] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.540470 16393 heartbeater.cc:503] Master 127.0.198.62:38905 was elected leader, sending a full tablet report...
I20260929 18:38:41.540614 16679 ts_tablet_manager.cc:1428] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 18:38:41.714166 16662 tablet.cc:2369] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:38:41.755818 16528 tablet.cc:2369] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:38:41.789786 16394 tablet.cc:2369] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:38:41.836284 16684 raft_consensus.cc:482] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:41.836514 16684 raft_consensus.cc:504] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.837702 16684 leader_election.cc:288] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053), a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715)
I20260929 18:38:41.841840 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.838008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:33715 (local address 127.0.198.2:59845)
0929 18:38:41.838402 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.838414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.838537 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.838799 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.838802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.838842 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:41.838999 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.839005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.839887 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.839889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.840688 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.840698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.841341 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.841344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.841465 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.841523 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.841593 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.841682 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":259,"spinlock_wait_cycles":1664,"thread_start_us":64,"threads_started":1}
I20260929 18:38:41.841816 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.838101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:35961 (local address 127.0.198.1:35053)
0929 18:38:41.838233 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.838238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.838478 (+ 240us) server_negotiation.cc:407] Connection header received
0929 18:38:41.838592 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.838595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.838648 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.838705 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:41.839063 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.839609 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.840752 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.841200 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.841298 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.841361 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.841619 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.841690 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:38:41.841822 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.838126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:59845 (local address 127.0.198.3:33715)
0929 18:38:41.838234 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.838238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.838516 (+ 278us) server_negotiation.cc:407] Connection header received
0929 18:38:41.838636 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.838638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.838705 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.838777 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:41.839147 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.839778 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.840823 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.841218 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.841317 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.841378 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.841621 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.841692 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:38:41.841920 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.838002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:35053 (local address 127.0.198.2:35961)
0929 18:38:41.838375 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.838391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:41.838499 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:41.838763 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:41.838766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:41.838783 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:41.838955 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.838961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.839732 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.839735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.840642 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:41.840652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.841345 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.841349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:41.841508 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.841526 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:38:41.841590 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:38:41.841687 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":46,"spinlock_wait_cycles":10624,"thread_start_us":99,"threads_started":1}
I20260929 18:38:41.842640 16615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc0d148124f466ca26792d4079a438d" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" is_pre_election: true
I20260929 18:38:41.842638 16347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc0d148124f466ca26792d4079a438d" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" is_pre_election: true
I20260929 18:38:41.842918 16347 raft_consensus.cc:2454] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 0.
I20260929 18:38:41.842923 16615 raft_consensus.cc:2454] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 0.
I20260929 18:38:41.843410 16417 leader_election.cc:302] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da3c2d0cdea4bb6b16db539dc5a4905, 5b7738d7ac7d4ad286863268d3f0ff25; no voters:
I20260929 18:38:41.843705 16684 raft_consensus.cc:2781] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:38:41.843786 16684 raft_consensus.cc:482] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:38:41.843818 16684 raft_consensus.cc:3037] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:41.844870 16684 raft_consensus.cc:504] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.845217 16684 leader_election.cc:288] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 election: Requested vote from peers 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053), a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715)
I20260929 18:38:41.845621 16615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc0d148124f466ca26792d4079a438d" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba"
I20260929 18:38:41.845753 16615 raft_consensus.cc:3037] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:41.845777 16347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc0d148124f466ca26792d4079a438d" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905"
I20260929 18:38:41.845903 16347 raft_consensus.cc:3037] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:41.846812 16615 raft_consensus.cc:2454] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 1.
I20260929 18:38:41.847045 16347 raft_consensus.cc:2454] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 1.
I20260929 18:38:41.847121 16415 leader_election.cc:302] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba; no voters:
I20260929 18:38:41.847306 16684 raft_consensus.cc:2781] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:41.847546 16684 raft_consensus.cc:686] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 LEADER]: Becoming Leader. State: Replica: 5b7738d7ac7d4ad286863268d3f0ff25, State: Running, Role: LEADER
I20260929 18:38:41.847904 16684 consensus_queue.cc:237] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:38:41.851078 16201 catalog_manager.cc:5495] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260929 18:38:41.879081 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.875551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.0.198.2:38317)
0929 18:38:41.875669 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:41.875673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:41.875686 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:41.875765 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:41.875767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:41.875820 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:41.875939 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:38:41.876224 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.876733 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.877665 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:41.877899 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:41.878012 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:41.878334 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:38:41.878754 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:38:41.878864 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:41.878913 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:38:41.878966 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:38:41.884874 16615 raft_consensus.cc:1261] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Refusing update from remote peer 5b7738d7ac7d4ad286863268d3f0ff25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:41.884868 16347 raft_consensus.cc:1261] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Refusing update from remote peer 5b7738d7ac7d4ad286863268d3f0ff25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:41.885531 16684 consensus_queue.cc:1035] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:41.885722 16689 consensus_queue.cc:1035] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:41.900965 16201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:41.901125 16201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:41.903594 16201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Sending DeleteTablet for 3 replicas of tablet fcc0d148124f466ca26792d4079a438d
I20260929 18:38:41.904412 16595 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc0d148124f466ca26792d4079a438d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:41 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:43324
I20260929 18:38:41.904405 16461 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc0d148124f466ca26792d4079a438d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:41 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:44250
I20260929 18:38:41.904950 16701 tablet_replica.cc:330] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: stopping tablet replica
I20260929 18:38:41.904976 16702 tablet_replica.cc:330] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: stopping tablet replica
I20260929 18:38:41.905002 16327 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc0d148124f466ca26792d4079a438d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:41 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33384
I20260929 18:38:41.905294 16701 raft_consensus.cc:2229] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:38:41.905340 16702 raft_consensus.cc:2229] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:41.905655 16701 raft_consensus.cc:2258] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:41.905730 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 16264
I20260929 18:38:41.905700 16702 raft_consensus.cc:2258] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:41.907527 16701 ts_tablet_manager.cc:1905] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:41.907527 16702 ts_tablet_manager.cc:1905] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:41.910271 16701 ts_tablet_manager.cc:1918] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:41.910365 16701 log.cc:1196] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/fcc0d148124f466ca26792d4079a438d
I20260929 18:38:41.910320 16702 ts_tablet_manager.cc:1918] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:41.910583 16702 log.cc:1196] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/fcc0d148124f466ca26792d4079a438d
I20260929 18:38:41.910646 16701 ts_tablet_manager.cc:1939] T fcc0d148124f466ca26792d4079a438d P 5b7738d7ac7d4ad286863268d3f0ff25: Deleting consensus metadata
I20260929 18:38:41.910962 16702 ts_tablet_manager.cc:1939] T fcc0d148124f466ca26792d4079a438d P a750095c0ccd49de8ef4f2d7dc8901ba: Deleting consensus metadata
I20260929 18:38:41.911902 16186 catalog_manager.cc:4841] TS 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317): tablet fcc0d148124f466ca26792d4079a438d (table test-table [id=2aa4888c10a54e5caabef3e75a2357e3]) successfully deleted
I20260929 18:38:41.912098 16187 catalog_manager.cc:4841] TS a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715): tablet fcc0d148124f466ca26792d4079a438d (table test-table [id=2aa4888c10a54e5caabef3e75a2357e3]) successfully deleted
W20260929 18:38:41.913210 16189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:35053 (error 108)
I20260929 18:38:41.913925 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 16398
I20260929 18:38:41.914330 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:41.914018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:35053 (local address 127.0.0.1:33390)
0929 18:38:41.914178 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:41.914240 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 18:38:41.915544 16189 catalog_manager.cc:4584] TS 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcc0d148124f466ca26792d4079a438d: Network error: Client connection negotiation failed: client connection to 127.0.198.1:35053: connect: Connection refused (error 111)
I20260929 18:38:41.918798 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 16532
I20260929 18:38:41.923454 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 16170
I20260929 18:39:01.928983 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38905
--webserver_interface=127.0.198.62
--webserver_port=38901
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:38905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:02.035975 16705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:02.036252 16705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:02.036293 16705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:02.040349 16705 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:39:02.040414 16705 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:39:02.040429 16705 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:39:02.040442 16705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:02.040459 16705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:02.040478 16705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:02.040506 16705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:02.040535 16705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:02.045930 16705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:38905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:02.047046 16705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:02.048051 16705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:02.060994 16711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:02.061003 16712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:02.061053 16705 server_base.cc:1034] running on GCE node
W20260929 18:39:02.061223 16714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:02.061815 16705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:02.063015 16705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:02.064186 16705 hybrid_clock.cc:648] HybridClock initialized: now 1790707142064159 us; error 42 us; skew 500 ppm
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:02.066983 16705 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:39:02.067931 16705 webserver.cc:460] Webserver started at http://127.0.198.62:38901/ using document root <none> and password file <none>
I20260929 18:39:02.068490 16705 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:02.068570 16705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:02.072618 16705 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:02.074921 16721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.076340 16705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:02.076483 16705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "43356f430fac4f4a85000ce1ec490e92"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "e28a2070f42930925c92955383da20ef"
server_key_iv: "d3c9c76a3a057543929fe09c5a1adf57"
server_key_version: "encryptionkey@0"
I20260929 18:39:02.077090 16705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:02.121021 16705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:02.124439 16705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:02.124632 16705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:02.132009 16705 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:38905
I20260929 18:39:02.132395 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:38905 every 8 connection(s)
I20260929 18:39:02.133246 16705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:02.135790 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 16705
I20260929 18:39:02.136286 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:35053
--local_ip_for_outbound_sockets=127.0.198.1
--tserver_master_addrs=127.0.198.62:38905
--webserver_port=41493
--webserver_interface=127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:02.138428 16774 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:39:02.140240 16774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Bootstrap starting.
I20260929 18:39:02.157043 16774 log.cc:824] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:02.165954 16774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:39:02.166357 16774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Bootstrap complete.
I20260929 18:39:02.170229 16774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:02.170651 16774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43356f430fac4f4a85000ce1ec490e92, State: Initialized, Role: FOLLOWER
I20260929 18:39:02.171111 16774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:02.171223 16774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:02.171291 16774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:02.171378 16774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:39:02.173223 16774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:02.173532 16774 leader_election.cc:302] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43356f430fac4f4a85000ce1ec490e92; no voters:
I20260929 18:39:02.173811 16774 leader_election.cc:288] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:39:02.173911 16778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:39:02.174180 16778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 LEADER]: Becoming Leader. State: Replica: 43356f430fac4f4a85000ce1ec490e92, State: Running, Role: LEADER
I20260929 18:39:02.174531 16778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:02.175093 16774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:02.175447 16778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43356f430fac4f4a85000ce1ec490e92. Latest consensus state: current_term: 2 leader_uuid: "43356f430fac4f4a85000ce1ec490e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } } }
I20260929 18:39:02.175555 16778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:02.176116 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43356f430fac4f4a85000ce1ec490e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } } }
I20260929 18:39:02.176220 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:02.176970 16782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:02.181545 16782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:02.182257 16782 catalog_manager.cc:1259] Loaded cluster ID: 496e6c74362241cda8a84b9697e8e3d3
I20260929 18:39:02.182319 16782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:02.184784 16782 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:02.185520 16782 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Loaded TSK: 0
I20260929 18:39:02.203961 16782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Generated new TSK 1
I20260929 18:39:02.204277 16782 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:39:02.249290 16776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:02.249570 16776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:02.249631 16776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:02.253796 16776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:02.253866 16776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:02.253948 16776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:02.259387 16776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:35053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=41493
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:02.260433 16776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:02.261425 16776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:02.271121 16801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:02.271144 16804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:02.271476 16776 server_base.cc:1034] running on GCE node
W20260929 18:39:02.271548 16802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:02.272022 16776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:02.272783 16776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:02.274011 16776 hybrid_clock.cc:648] HybridClock initialized: now 1790707142273976 us; error 45 us; skew 500 ppm
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:02.277508 16776 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:39:02.278481 16776 webserver.cc:460] Webserver started at http://127.0.198.1:41493/ using document root <none> and password file <none>
I20260929 18:39:02.279021 16776 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:02.279103 16776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:02.283563 16776 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:02.285938 16811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.287345 16776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:02.287472 16776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "2da3c2d0cdea4bb6b16db539dc5a4905"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "797f96685fed95abce93425a1898b419"
server_key_iv: "636cbf56aa33cf963926ac33319135ac"
server_key_version: "encryptionkey@0"
I20260929 18:39:02.288074 16776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:02.303387 16776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:02.306522 16776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:02.306710 16776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:02.307288 16776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:02.308357 16818 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:39:02.313676 16776 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:39:02.313755 16776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 18:39:02.313820 16776 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:39:02.315502 16776 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:39:02.315548 16776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:39:02.315899 16818 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap starting.
I20260929 18:39:02.325320 16776 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:35053
I20260929 18:39:02.325567 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:35053 every 8 connection(s)
I20260929 18:39:02.326370 16776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:02.332909 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 16776
I20260929 18:39:02.333348 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:38317
--local_ip_for_outbound_sockets=127.0.198.2
--tserver_master_addrs=127.0.198.62:38905
--webserver_port=34983
--webserver_interface=127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:02.335879 16818 log.cc:824] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:02.343569 16818 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:39:02.343968 16818 tablet_bootstrap.cc:492] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap complete.
I20260929 18:39:02.344339 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.328406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:42165)
0929 18:39:02.329154 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.329208 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.330022 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.331130 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.331139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.331558 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:02.332030 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.332044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.333658 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.333662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.334953 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.334964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.335663 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.335667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.335846 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.336460 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:02.336480 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:02.338284 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:02.340852 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.340858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.340871 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.341208 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.341627 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.341630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.341632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.341787 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.342533 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:02.342539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:02.342855 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.343035 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.343370 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":132,"threads_started":1}
I20260929 18:39:02.344993 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.328378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:42165 (local address 127.0.198.62:38905)
0929 18:39:02.328879 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.328898 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.330061 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 18:39:02.330436 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.330446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.330746 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.331047 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:02.332209 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.333483 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.335073 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.335529 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.335698 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.338507 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:02.338515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:02.338534 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:02.338570 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:02.340710 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.341342 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.341349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.341355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.341519 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.341908 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.341912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.341914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.342414 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:02.342555 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.343220 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.343531 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":105,"threads_started":1}
I20260929 18:39:02.346215 16818 ts_tablet_manager.cc:1397] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.003s
I20260929 18:39:02.347343 16926 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38905
I20260929 18:39:02.347601 16926 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:02.348132 16926 heartbeater.cc:511] Master 127.0.198.62:38905 requested a full tablet report, sending...
I20260929 18:39:02.348806 16818 raft_consensus.cc:348] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:39:02.349196 16818 raft_consensus.cc:729] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da3c2d0cdea4bb6b16db539dc5a4905, State: Initialized, Role: FOLLOWER
I20260929 18:39:02.349675 16818 consensus_queue.cc:260] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } }
I20260929 18:39:02.350701 16818 ts_tablet_manager.cc:1428] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260929 18:39:02.350936 16738 ts_manager.cc:194] Registered new tserver with Master: 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053)
I20260929 18:39:02.354182 16738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:42165
I20260929 18:39:02.356552 16926 heartbeater.cc:503] Master 127.0.198.62:38905 was elected leader, sending a full tablet report...
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.1@KRBTEST.COM
I20260929 18:39:02.364068 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.353685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:35053 (local address 127.0.0.1:57700)
0929 18:39:02.354421 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.354463 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.354598 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.355082 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.355088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.355523 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:02.355645 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.355654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.357022 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.357026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.357911 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.357921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.358643 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.358646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.358725 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.359236 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:02.359248 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:02.360589 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:02.362540 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.362547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.362551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.362759 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.363033 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.363035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.363036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.363103 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.363578 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:02.363584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:02.363685 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.363790 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.363939 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":121,"threads_started":1}
I20260929 18:39:02.364421 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.353769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.198.1:35053)
0929 18:39:02.354164 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.354176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.354567 (+ 391us) server_negotiation.cc:407] Connection header received
0929 18:39:02.354732 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.354738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.354940 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.355070 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:02.355767 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.356921 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.358052 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.358533 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.358635 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.360717 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:02.360727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:02.360739 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:02.360764 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:02.362446 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.362856 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.362862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.362864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.362954 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.363181 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.363184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.363186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.363473 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:02.363589 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.363884 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.364041 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":103,"threads_started":1}
I20260929 18:39:02.365331 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc0d148124f466ca26792d4079a438d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:41 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:57700
I20260929 18:39:02.366178 16935 tablet_replica.cc:330] stopping tablet replica
I20260929 18:39:02.366447 16935 raft_consensus.cc:2229] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:02.366669 16935 raft_consensus.cc:2258] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:02.367594 16935 ts_tablet_manager.cc:1905] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:02.371021 16935 ts_tablet_manager.cc:1918] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:02.371100 16935 log.cc:1196] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal/wals/fcc0d148124f466ca26792d4079a438d
I20260929 18:39:02.371317 16935 ts_tablet_manager.cc:1939] T fcc0d148124f466ca26792d4079a438d P 2da3c2d0cdea4bb6b16db539dc5a4905: Deleting consensus metadata
I20260929 18:39:02.373217 16725 catalog_manager.cc:4841] TS 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053): tablet fcc0d148124f466ca26792d4079a438d (table test-table [id=2aa4888c10a54e5caabef3e75a2357e3]) successfully deleted
W20260929 18:39:02.449549 16930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:02.449841 16930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:02.449903 16930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:02.454002 16930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:02.454063 16930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:02.454145 16930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:02.459424 16930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:38317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=34983
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:02.460484 16930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:02.461493 16930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:02.471216 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:02.471201 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:02.471321 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:02.471437 16930 server_base.cc:1034] running on GCE node
I20260929 18:39:02.471997 16930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:02.472682 16930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:02.473894 16930 hybrid_clock.cc:648] HybridClock initialized: now 1790707142473883 us; error 26 us; skew 500 ppm
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:02.477037 16930 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:39:02.478050 16930 webserver.cc:460] Webserver started at http://127.0.198.2:34983/ using document root <none> and password file <none>
I20260929 18:39:02.478595 16930 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:02.478663 16930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:02.482642 16930 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:02.484670 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.485910 16930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:02.486048 16930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "5b7738d7ac7d4ad286863268d3f0ff25"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "926f4e859363a2d974510f161ef5ed11"
server_key_iv: "2bfa37bc1c23c4280f644a1a1ff146a1"
server_key_version: "encryptionkey@0"
I20260929 18:39:02.486575 16930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:02.501688 16930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:02.504863 16930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:02.505044 16930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:02.505616 16930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:02.506445 16930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:02.506491 16930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.506543 16930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:02.506572 16930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.514863 16930 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:38317
I20260929 18:39:02.514891 17063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:38317 every 8 connection(s)
I20260929 18:39:02.515805 16930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:02.520481 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 16930
I20260929 18:39:02.520891 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:33715
--local_ip_for_outbound_sockets=127.0.198.3
--tserver_master_addrs=127.0.198.62:38905
--webserver_port=44933
--webserver_interface=127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:02.530153 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.517682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:51693 (local address 127.0.198.62:38905)
0929 18:39:02.517831 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.517835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.518757 (+ 922us) server_negotiation.cc:407] Connection header received
0929 18:39:02.520102 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.520105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.520158 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.520379 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:02.521698 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.522290 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.523449 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.523684 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.523771 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.526542 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:02.526546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:02.526548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:02.526574 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:02.528053 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.528597 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.528600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.528601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.528684 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.529003 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.529006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.529008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.529203 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:02.529288 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.529818 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.529962 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:39:02.530994 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.518114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:51693)
0929 18:39:02.518598 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.518645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.519865 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.520562 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.520571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.521004 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:02.521541 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.521555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.522411 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.522414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.523320 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.523330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.523829 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.523833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.523997 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.524688 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:02.524704 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:02.526369 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:02.528179 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.528186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.528205 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.528496 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.528768 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.528771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.528773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.528915 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.529303 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:02.529310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:02.529562 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.529758 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.530139 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":119,"threads_started":1}
I20260929 18:39:02.532172 17064 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38905
I20260929 18:39:02.532433 17064 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:02.532986 17064 heartbeater.cc:511] Master 127.0.198.62:38905 requested a full tablet report, sending...
I20260929 18:39:02.534163 16738 ts_manager.cc:194] Registered new tserver with Master: 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317)
I20260929 18:39:02.534915 16738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:51693
W20260929 18:39:02.631376 17067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:02.631639 17067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:02.631681 17067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:02.635659 17067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:02.635723 17067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:02.635780 17067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:02.641043 17067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:33715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=44933
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:02.642190 17067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:02.643216 17067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:02.652729 17072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:02.652850 17067 server_base.cc:1034] running on GCE node
W20260929 18:39:02.652853 17075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:02.652732 17073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:02.653362 17067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:02.654016 17067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:02.655205 17067 hybrid_clock.cc:648] HybridClock initialized: now 1790707142655182 us; error 31 us; skew 500 ppm
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:02.658378 17067 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:39:02.659340 17067 webserver.cc:460] Webserver started at http://127.0.198.3:44933/ using document root <none> and password file <none>
I20260929 18:39:02.659904 17067 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:02.659986 17067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:02.664000 17067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:02.666260 17082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.667486 17067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:39:02.667594 17067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "a750095c0ccd49de8ef4f2d7dc8901ba"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "e977599c18c3ec73368ec4cf3fe5ee37"
server_key_iv: "b0060a220de6f22c3702940ac91582d4"
server_key_version: "encryptionkey@0"
I20260929 18:39:02.668139 17067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:02.686945 17067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:02.690410 17067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:02.690585 17067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:02.691146 17067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:02.691960 17067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:02.692005 17067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.692044 17067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:02.692075 17067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:02.700819 17067 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:33715
I20260929 18:39:02.700873 17195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:33715 every 8 connection(s)
I20260929 18:39:02.701840 17067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:02.707837 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 17067
Sep 29 18:39:02 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:02.715772 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.708453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.0.198.62:38905)
0929 18:39:02.708673 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.708677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.708824 (+ 147us) server_negotiation.cc:407] Connection header received
0929 18:39:02.708932 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.708935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.708977 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.709045 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:02.709790 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.710329 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.711155 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.711371 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.711453 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.712721 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:02.712725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:02.712728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:02.712766 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:02.714298 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.714705 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.714707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.714708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.714772 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.715015 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.715017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.715019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.715173 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:02.715391 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.715534 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.715628 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260929 18:39:02.716019 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.703875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:36321 (local address 127.0.198.62:38905)
0929 18:39:02.704000 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.704004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.704851 (+ 847us) server_negotiation.cc:407] Connection header received
0929 18:39:02.706162 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.706166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.706225 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.706359 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:02.707539 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.708272 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.709599 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.709851 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.709940 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.712294 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:02.712297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:02.712299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:02.712321 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:02.714000 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.714504 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.714507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.714508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.714575 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.714897 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.714899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.714900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.715090 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:02.715181 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.715800 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.715901 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:39:02.717033 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.704218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:36321)
0929 18:39:02.704682 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.704725 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.705923 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.706553 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.706560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.706921 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:02.707391 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.707406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.708404 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.708408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.709474 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.709485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.710022 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.710027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.710187 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.710734 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:02.710750 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:02.712136 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:02.714111 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.714116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.714130 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.714411 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.714653 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:02.714656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:02.714658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:02.714810 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:02.715196 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:02.715201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:02.715539 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.715763 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.716073 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260929 18:39:02.718102 17196 heartbeater.cc:347] Connected to a master server at 127.0.198.62:38905
I20260929 18:39:02.718379 17196 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:02.718847 17196 heartbeater.cc:511] Master 127.0.198.62:38905 requested a full tablet report, sending...
I20260929 18:39:02.719877 16738 ts_manager.cc:194] Registered new tserver with Master: a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715)
I20260929 18:39:02.720499 16738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:36321
I20260929 18:39:02.729535 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:02.733675 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.730217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.0.198.62:38905)
0929 18:39:02.730330 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.730335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.730540 (+ 205us) server_negotiation.cc:407] Connection header received
0929 18:39:02.730702 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.730704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.730809 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.730900 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:02.731146 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.731704 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.732573 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.732782 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.732865 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.733159 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:02.733304 (+ 145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 18:39:02.733547 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44902: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 18:39:02.733740 16928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44902: token expired
W20260929 18:39:02.734081 16666 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.198.62:38905)
I20260929 18:39:02.740787 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.734390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.0.198.62:38905)
0929 18:39:02.734511 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.734514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.734526 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:39:02.734599 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.734601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.734637 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.734698 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:02.735229 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.735699 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.736551 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.736752 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.736832 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.738029 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:02.738032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:02.738034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:02.738061 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:02.739367 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.739789 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.739792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.739793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.739875 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:02.740088 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:02.740090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:02.740091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:02.740255 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:02.740353 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.740536 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.740627 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:39:02.743120 16738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:02.746313 16738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:02.754524 16860 tablet_service.cc:1467] Processing CreateTablet for tablet 06cb40df419f404ea38a558f88d48b34 (DEFAULT_TABLE table=test-table [id=e06715c53e22442c9621f3e0c2da9523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:02.755136 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cb40df419f404ea38a558f88d48b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:02.757843 16818 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap starting.
I20260929 18:39:02.758667 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.753331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:38317 (local address 127.0.0.1:37474)
0929 18:39:02.753587 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.753600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.753687 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.754656 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.754659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.754678 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:02.754773 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.754779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.756216 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.756220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.757120 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.757130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.758220 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.758224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.758321 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.758370 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.758447 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.758525 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":118,"threads_started":1}
I20260929 18:39:02.758742 16818 tablet_bootstrap.cc:654] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:02.758839 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.753067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:33715 (local address 127.0.0.1:33772)
0929 18:39:02.753197 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.753220 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.753299 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.754180 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.754183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.754202 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:02.754325 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.754332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.755896 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.755900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.757076 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.757086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.758481 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.758485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.758580 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.758603 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.758650 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.758715 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:39:02.759095 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.753783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.198.2:38317)
0929 18:39:02.754198 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.754209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.754230 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:39:02.754315 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.754320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.754515 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.754664 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:02.754867 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.756101 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.757255 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.758088 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.758199 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.758373 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.758839 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.758949 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":138,"threads_started":1}
I20260929 18:39:02.759569 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.753251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.0.198.3:33715)
0929 18:39:02.753616 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.753628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.753652 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:39:02.753742 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.753747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.754028 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.754201 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:02.754419 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.755776 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.757207 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.758333 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.759141 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.759289 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.759342 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.759434 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":161,"threads_started":1}
I20260929 18:39:02.760829 16818 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: No bootstrap required, opened a new log
I20260929 18:39:02.760922 16818 ts_tablet_manager.cc:1397] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:02.761348 16818 raft_consensus.cc:348] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.761469 16818 raft_consensus.cc:374] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:02.761513 16818 raft_consensus.cc:729] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da3c2d0cdea4bb6b16db539dc5a4905, State: Initialized, Role: FOLLOWER
I20260929 18:39:02.761637 16818 consensus_queue.cc:260] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.761933 16818 ts_tablet_manager.cc:1428] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:39:02.762787 17130 tablet_service.cc:1467] Processing CreateTablet for tablet 06cb40df419f404ea38a558f88d48b34 (DEFAULT_TABLE table=test-table [id=e06715c53e22442c9621f3e0c2da9523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:02.763765 17130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cb40df419f404ea38a558f88d48b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:02.767975 16998 tablet_service.cc:1467] Processing CreateTablet for tablet 06cb40df419f404ea38a558f88d48b34 (DEFAULT_TABLE table=test-table [id=e06715c53e22442c9621f3e0c2da9523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:02.768858 16998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cb40df419f404ea38a558f88d48b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:02.770429 17209 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Bootstrap starting.
I20260929 18:39:02.773313 17209 tablet_bootstrap.cc:654] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:02.774236 17209 log.cc:824] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:02.776436 17209 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: No bootstrap required, opened a new log
I20260929 18:39:02.776633 17209 ts_tablet_manager.cc:1397] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:39:02.777000 17211 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Bootstrap starting.
I20260929 18:39:02.779387 17211 tablet_bootstrap.cc:654] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:02.779712 17209 raft_consensus.cc:348] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.779943 17209 raft_consensus.cc:374] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:02.779991 17209 raft_consensus.cc:729] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a750095c0ccd49de8ef4f2d7dc8901ba, State: Initialized, Role: FOLLOWER
I20260929 18:39:02.780125 17211 log.cc:824] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:02.780435 17209 consensus_queue.cc:260] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.781097 17196 heartbeater.cc:503] Master 127.0.198.62:38905 was elected leader, sending a full tablet report...
I20260929 18:39:02.781385 17209 ts_tablet_manager.cc:1428] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:39:02.782217 17211 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: No bootstrap required, opened a new log
I20260929 18:39:02.782415 17211 ts_tablet_manager.cc:1397] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:39:02.785787 17211 raft_consensus.cc:348] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.786073 17211 raft_consensus.cc:374] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:02.786130 17211 raft_consensus.cc:729] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7738d7ac7d4ad286863268d3f0ff25, State: Initialized, Role: FOLLOWER
I20260929 18:39:02.786537 17211 consensus_queue.cc:260] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.787241 17064 heartbeater.cc:503] Master 127.0.198.62:38905 was elected leader, sending a full tablet report...
I20260929 18:39:02.787478 17211 ts_tablet_manager.cc:1428] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:39:02.804059 17214 raft_consensus.cc:482] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:02.804255 17214 raft_consensus.cc:504] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.805358 17214 leader_election.cc:288] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715), 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053)
I20260929 18:39:02.809303 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.805581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:33715 (local address 127.0.198.2:60275)
0929 18:39:02.805725 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.805740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.805826 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.806154 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.806157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.806178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:02.806301 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.806307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.807120 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.807124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.808042 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.808051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.808864 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.808868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.808978 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.809003 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.809087 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.809161 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":21}
I20260929 18:39:02.809357 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.805650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:60275 (local address 127.0.198.3:33715)
0929 18:39:02.805773 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.805779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.805792 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:39:02.805998 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.806001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.806058 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.806130 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:02.806421 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.807013 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.808172 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.808730 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.808809 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.808867 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.809111 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.809168 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:39:02.809909 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.805608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:35053 (local address 127.0.198.2:44307)
0929 18:39:02.806010 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.806024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:02.806118 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:02.806470 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:02.806473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:02.806490 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:02.806591 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.806598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.807458 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.807461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.808417 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:02.808428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.809241 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.809245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:02.809339 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.809656 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 18:39:02.809732 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:39:02.809809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260929 18:39:02.809957 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.805897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:44307 (local address 127.0.198.1:35053)
0929 18:39:02.806029 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.806032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.806079 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:39:02.806278 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.806283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.806352 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.806444 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:02.806690 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.807347 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.808557 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.809130 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.809305 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.809473 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.809770 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.809849 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:39:02.810014 17141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cb40df419f404ea38a558f88d48b34" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" is_pre_election: true
I20260929 18:39:02.810401 17141 raft_consensus.cc:2454] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 0.
I20260929 18:39:02.810557 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cb40df419f404ea38a558f88d48b34" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" is_pre_election: true
I20260929 18:39:02.810882 16880 raft_consensus.cc:2454] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 0.
I20260929 18:39:02.810899 16952 leader_election.cc:302] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba; no voters:
I20260929 18:39:02.811168 17214 raft_consensus.cc:2781] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:02.811278 17214 raft_consensus.cc:482] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:02.811331 17214 raft_consensus.cc:3037] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:02.812351 17214 raft_consensus.cc:504] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.812657 17214 leader_election.cc:288] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 election: Requested vote from peers a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715), 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053)
I20260929 18:39:02.813024 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cb40df419f404ea38a558f88d48b34" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905"
I20260929 18:39:02.813033 17141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cb40df419f404ea38a558f88d48b34" candidate_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba"
I20260929 18:39:02.813156 17141 raft_consensus.cc:3037] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:02.813156 16880 raft_consensus.cc:3037] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:02.814406 17141 raft_consensus.cc:2454] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 1.
I20260929 18:39:02.814406 16880 raft_consensus.cc:2454] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7738d7ac7d4ad286863268d3f0ff25 in term 1.
I20260929 18:39:02.814710 16952 leader_election.cc:302] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba; no voters:
I20260929 18:39:02.814975 17214 raft_consensus.cc:2781] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:02.815220 17214 raft_consensus.cc:686] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 LEADER]: Becoming Leader. State: Replica: 5b7738d7ac7d4ad286863268d3f0ff25, State: Running, Role: LEADER
I20260929 18:39:02.815574 17214 consensus_queue.cc:237] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:02.818933 16737 catalog_manager.cc:5495] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } health_report { overall_health: UNKNOWN } } }
W20260929 18:39:02.827770 16927 tablet.cc:2369] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:02.840085 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:02.836272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.0.198.2:38317)
0929 18:39:02.836392 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:02.836395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:02.836408 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:39:02.836507 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:02.836509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:02.836565 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:02.836652 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:02.836922 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.837444 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.838446 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:02.838662 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:02.838754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:02.839111 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:02.839585 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:02.839678 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:02.839860 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:39:02.839921 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:39:02.846042 16880 raft_consensus.cc:1261] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Refusing update from remote peer 5b7738d7ac7d4ad286863268d3f0ff25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:02.846051 17141 raft_consensus.cc:1261] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Refusing update from remote peer 5b7738d7ac7d4ad286863268d3f0ff25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:02.846733 17216 consensus_queue.cc:1035] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:02.846912 17214 consensus_queue.cc:1035] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:02.852375 17218 mvcc.cc:205] Tried to move back new op lower bound from 7334736457089806336 to 7334736456975773696. Current Snapshot: MvccSnapshot[applied={T|T < 7334736457089806336}]
I20260929 18:39:02.861603 16738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:02.861793 16738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:02.864076 16738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Sending DeleteTablet for 3 replicas of tablet 06cb40df419f404ea38a558f88d48b34
I20260929 18:39:02.864702 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet 06cb40df419f404ea38a558f88d48b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:02 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:57700
I20260929 18:39:02.864769 17130 tablet_service.cc:1514] Processing DeleteTablet for tablet 06cb40df419f404ea38a558f88d48b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:02 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:33772
I20260929 18:39:02.864789 16998 tablet_service.cc:1514] Processing DeleteTablet for tablet 06cb40df419f404ea38a558f88d48b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:02 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:37474
I20260929 18:39:02.864948 16935 tablet_replica.cc:330] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: stopping tablet replica
I20260929 18:39:02.865068 16935 raft_consensus.cc:2229] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:02.865196 16935 raft_consensus.cc:2258] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:02.865414 17229 tablet_replica.cc:330] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: stopping tablet replica
I20260929 18:39:02.865443 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 16776
I20260929 18:39:02.865696 17229 raft_consensus.cc:2229] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:39:02.866043 17229 raft_consensus.cc:2258] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:02.867610 17229 ts_tablet_manager.cc:1905] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:02.870430 17229 ts_tablet_manager.cc:1918] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:02.870533 17229 log.cc:1196] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/06cb40df419f404ea38a558f88d48b34
I20260929 18:39:02.870850 17229 ts_tablet_manager.cc:1939] T 06cb40df419f404ea38a558f88d48b34 P 5b7738d7ac7d4ad286863268d3f0ff25: Deleting consensus metadata
W20260929 18:39:02.871640 16725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:35053 (error 108)
I20260929 18:39:02.872180 16722 catalog_manager.cc:4841] TS 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317): tablet 06cb40df419f404ea38a558f88d48b34 (table test-table [id=e06715c53e22442c9621f3e0c2da9523]) successfully deleted
W20260929 18:39:02.872958 16933 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:39:02.872661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:35053 (local address 127.0.0.1:57716)
0929 18:39:02.872812 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:02.872878 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:35053: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 18:39:02.873121 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 16930
W20260929 18:39:02.874174 16725 catalog_manager.cc:4584] TS 2da3c2d0cdea4bb6b16db539dc5a4905 (127.0.198.1:35053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06cb40df419f404ea38a558f88d48b34: Network error: Client connection negotiation failed: client connection to 127.0.198.1:35053: connect: Connection reset by peer (error 104)
I20260929 18:39:02.874802 17228 tablet_replica.cc:330] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: stopping tablet replica
I20260929 18:39:02.875097 17228 raft_consensus.cc:2229] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:02.875418 17228 raft_consensus.cc:2258] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:02.876753 17228 ts_tablet_manager.cc:1905] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:02.878921 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 17067
I20260929 18:39:02.883972 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 16705
I20260929 18:39:22.889853 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38905
--webserver_interface=127.0.198.62
--webserver_port=38901
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:38905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:22.999191 17231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:22.999490 17231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:22.999544 17231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:23.003633 17231 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:39:23.003700 17231 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:39:23.003719 17231 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:39:23.003738 17231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:23.003760 17231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:23.003784 17231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:23.003810 17231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:23.003840 17231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:23.009222 17231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:38905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:38905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:23.010360 17231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:23.011399 17231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:23.020537 17239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:23.020517 17236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:23.020632 17237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:23.020820 17231 server_base.cc:1034] running on GCE node
I20260929 18:39:23.021389 17231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:23.022573 17231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:23.023782 17231 hybrid_clock.cc:648] HybridClock initialized: now 1790707163023764 us; error 36 us; skew 500 ppm
Sep 29 18:39:23 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:23.026595 17231 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:39:23.027509 17231 webserver.cc:460] Webserver started at http://127.0.198.62:38901/ using document root <none> and password file <none>
I20260929 18:39:23.028046 17231 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:23.028112 17231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:23.031988 17231 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:23.034232 17246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:23.035501 17231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:23.035619 17231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "43356f430fac4f4a85000ce1ec490e92"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "e28a2070f42930925c92955383da20ef"
server_key_iv: "d3c9c76a3a057543929fe09c5a1adf57"
server_key_version: "encryptionkey@0"
I20260929 18:39:23.036157 17231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:23.070060 17231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:23.073130 17231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:23.073324 17231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:23.080627 17231 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:38905
I20260929 18:39:23.080873 17298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:38905 every 8 connection(s)
I20260929 18:39:23.081763 17231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:23.086251 17299 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:39:23.086721 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 17231
I20260929 18:39:23.087137 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:35053
--local_ip_for_outbound_sockets=127.0.198.1
--tserver_master_addrs=127.0.198.62:38905
--webserver_port=41493
--webserver_interface=127.0.198.1
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:23.088179 17299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Bootstrap starting.
I20260929 18:39:23.110896 17299 log.cc:824] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:23.122051 17299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:39:23.122428 17299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Bootstrap complete.
I20260929 18:39:23.126276 17299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:23.126680 17299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43356f430fac4f4a85000ce1ec490e92, State: Initialized, Role: FOLLOWER
I20260929 18:39:23.127171 17299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:23.127293 17299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:23.127359 17299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:23.127452 17299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:39:23.129349 17299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:23.129668 17299 leader_election.cc:302] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43356f430fac4f4a85000ce1ec490e92; no voters:
I20260929 18:39:23.129961 17299 leader_election.cc:288] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:39:23.130089 17303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:39:23.130282 17303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [term 3 LEADER]: Becoming Leader. State: Replica: 43356f430fac4f4a85000ce1ec490e92, State: Running, Role: LEADER
I20260929 18:39:23.130573 17303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } }
I20260929 18:39:23.131210 17299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:23.131525 17303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43356f430fac4f4a85000ce1ec490e92. Latest consensus state: current_term: 3 leader_uuid: "43356f430fac4f4a85000ce1ec490e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } } }
I20260929 18:39:23.131731 17303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:23.132829 17309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:23.133056 17304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "43356f430fac4f4a85000ce1ec490e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43356f430fac4f4a85000ce1ec490e92" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 38905 } } }
I20260929 18:39:23.133267 17304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:23.137795 17309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:23.138552 17309 catalog_manager.cc:1259] Loaded cluster ID: 496e6c74362241cda8a84b9697e8e3d3
I20260929 18:39:23.138607 17309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:23.141144 17309 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:23.141898 17309 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Loaded TSK: 0
I20260929 18:39:23.142162 17309 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43356f430fac4f4a85000ce1ec490e92: Loaded TSK: 1
F20260929 18:39:23.142987 17309 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd797d429ad google::LogMessage::Fail() at ??:0
@ 0x7fd797d4588a google::LogMessage::SendToLog() at ??:0
@ 0x7fd797d424de google::LogMessage::Flush() at ??:0
@ 0x7fd797d43a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd79aa0618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd79a9fef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd7a26a77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd7a268199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd7a2682178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd7a26c3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd7a26e282b std::function<>::operator()() at ??:0
@ 0x7fd7a2681b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd7a2682f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd7a267f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd7a26c3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd799f943d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd799f94c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd799f963f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd799f85512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd79b5c96db start_thread at ??:0
@ 0x7fd79684061f clone at ??:0
W20260929 18:39:23.223295 17301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:23.223534 17301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:23.223577 17301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:23.227911 17301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:23.227962 17301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:23.228022 17301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:23.233847 17301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:35053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=41493
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:23.234959 17301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:23.236073 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:23.245736 17346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:23.245633 17347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:23.246037 17301 server_base.cc:1034] running on GCE node
W20260929 18:39:23.246235 17349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:23.246690 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:23.247555 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:23.248771 17301 hybrid_clock.cc:648] HybridClock initialized: now 1790707163248743 us; error 46 us; skew 500 ppm
Sep 29 18:39:23 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:23.252486 17301 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:39:23.253772 17301 webserver.cc:460] Webserver started at http://127.0.198.1:41493/ using document root <none> and password file <none>
I20260929 18:39:23.254606 17301 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:23.254699 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:23.259940 17301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:39:23.262706 17356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:23.264359 17301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:23.264488 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "2da3c2d0cdea4bb6b16db539dc5a4905"
format_stamp: "Formatted at 2026-09-29 18:38:40 on dist-test-slave-q5qx"
server_key: "797f96685fed95abce93425a1898b419"
server_key_iv: "636cbf56aa33cf963926ac33319135ac"
server_key_version: "encryptionkey@0"
I20260929 18:39:23.265228 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:23.288558 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:23.292872 17301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:23.293092 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:23.293886 17301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:23.304713 17363 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:39:23.309232 17301 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:39:23.309340 17301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260929 18:39:23.309417 17301 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:39:23.311792 17301 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:39:23.311865 17301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260929 18:39:23.312278 17363 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap starting.
I20260929 18:39:23.322901 17301 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:35053
I20260929 18:39:23.323257 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:35053 every 8 connection(s)
I20260929 18:39:23.324348 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:23.324934 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 17301
I20260929 18:39:23.325419 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:38317
--local_ip_for_outbound_sockets=127.0.198.2
--tserver_master_addrs=127.0.198.62:38905
--webserver_port=34983
--webserver_interface=127.0.198.2
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:23.334898 17363 log.cc:824] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:23.342597 17363 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:39:23.342983 17363 tablet_bootstrap.cc:492] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Bootstrap complete.
I20260929 18:39:23.343901 17363 ts_tablet_manager.cc:1397] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.009s
I20260929 18:39:23.346354 17363 raft_consensus.cc:348] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:23.346788 17363 raft_consensus.cc:729] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da3c2d0cdea4bb6b16db539dc5a4905, State: Initialized, Role: FOLLOWER
I20260929 18:39:23.347270 17363 consensus_queue.cc:260] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:23.348246 17363 ts_tablet_manager.cc:1428] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260929 18:39:23.418658 17474 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:39:23.326891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:60565)
0929 18:39:23.328171 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.328233 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:23.329311 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:23.417792 (+ 88481us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":154,"threads_started":1}
W20260929 18:39:23.419134 17471 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:38905 (0 consecutive failures): Network error: Failed to ping master at 127.0.198.62:38905: Client connection negotiation failed: client connection to 127.0.198.62:38905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:39:23.419803 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.419561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:46015)
0929 18:39:23.419685 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.419749 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:39:23.420436 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.420236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:48815)
0929 18:39:23.420333 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.420390 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:39:23.420584 17471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:39:23.469697 17473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:23.470041 17473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:23.470110 17473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:23.474375 17473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:23.474448 17473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:23.474530 17473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:23.480252 17473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:38317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=34983
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:23.481400 17473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:23.482467 17473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:23.492210 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:23.492271 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:23.492398 17473 server_base.cc:1034] running on GCE node
W20260929 18:39:23.492205 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:23.492913 17473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:23.493541 17473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:23.494732 17473 hybrid_clock.cc:648] HybridClock initialized: now 1790707163494702 us; error 46 us; skew 500 ppm
Sep 29 18:39:23 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:23.497541 17473 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:39:23.498593 17473 webserver.cc:460] Webserver started at http://127.0.198.2:34983/ using document root <none> and password file <none>
I20260929 18:39:23.499158 17473 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:23.499229 17473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:23.503305 17473 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:39:23.505795 17491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:23.507159 17473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:23.507294 17473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "5b7738d7ac7d4ad286863268d3f0ff25"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "926f4e859363a2d974510f161ef5ed11"
server_key_iv: "2bfa37bc1c23c4280f644a1a1ff146a1"
server_key_version: "encryptionkey@0"
I20260929 18:39:23.507848 17473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:23.525336 17473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:23.528556 17473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:23.528736 17473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:23.529398 17473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:23.530251 17473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:23.530297 17473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:23.530350 17473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:23.530378 17473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:23.539610 17473 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:38317
I20260929 18:39:23.539669 17604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:38317 every 8 connection(s)
I20260929 18:39:23.540880 17473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:23.544981 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 17473
I20260929 18:39:23.545419 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.543329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:47209)
0929 18:39:23.543874 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.544423 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
I20260929 18:39:23.545511 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:33715
--local_ip_for_outbound_sockets=127.0.198.3
--tserver_master_addrs=127.0.198.62:38905
--webserver_port=44933
--webserver_interface=127.0.198.3
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:23.546327 17605 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:38905 (0 consecutive failures): Network error: Failed to ping master at 127.0.198.62:38905: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
I20260929 18:39:23.546993 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.546779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:50453)
0929 18:39:23.546884 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.546936 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:39:23.547659 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.547470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:38079)
0929 18:39:23.547563 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.547608 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:39:23.547799 17605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:39:23.657572 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:23.657852 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:23.657896 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:23.661893 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:23.661958 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:23.662053 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:23.667415 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:33715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=44933
--tserver_master_addrs=127.0.198.62:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:23.668432 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:23.669385 17608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:23.678840 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:23.678831 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:23.679054 17608 server_base.cc:1034] running on GCE node
W20260929 18:39:23.678835 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:23.679625 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:23.680269 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:23.681427 17608 hybrid_clock.cc:648] HybridClock initialized: now 1790707163681392 us; error 47 us; skew 500 ppm
Sep 29 18:39:23 dist-test-slave-q5qx krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:23.683977 17608 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:39:23.684842 17608 webserver.cc:460] Webserver started at http://127.0.198.3:44933/ using document root <none> and password file <none>
I20260929 18:39:23.685364 17608 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:23.685426 17608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:23.689445 17608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:23.691768 17623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:23.693045 17608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:23.693168 17608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "a750095c0ccd49de8ef4f2d7dc8901ba"
format_stamp: "Formatted at 2026-09-29 18:38:41 on dist-test-slave-q5qx"
server_key: "e977599c18c3ec73368ec4cf3fe5ee37"
server_key_iv: "b0060a220de6f22c3702940ac91582d4"
server_key_version: "encryptionkey@0"
I20260929 18:39:23.693753 17608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:23.717517 17608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:23.720635 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:23.720811 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:23.721393 17608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:23.722522 17630 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 18:39:23.726428 17630 ts_tablet_manager.cc:1852] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 18:39:23.726821 17630 ts_tablet_manager.cc:1905] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:23.730652 17630 ts_tablet_manager.cc:1918] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:23.730796 17630 log.cc:1196] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/06cb40df419f404ea38a558f88d48b34
I20260929 18:39:23.731104 17630 ts_tablet_manager.cc:1939] T 06cb40df419f404ea38a558f88d48b34 P a750095c0ccd49de8ef4f2d7dc8901ba: Deleting consensus metadata
I20260929 18:39:23.731376 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 18:39:23.731448 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260929 18:39:23.731515 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:23.731540 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:23.740168 17608 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:33715
I20260929 18:39:23.740253 17737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:33715 every 8 connection(s)
I20260929 18:39:23.741057 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:23.742408 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 17608
I20260929 18:39:23.744756 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.743385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:50081)
0929 18:39:23.743858 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.743965 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
W20260929 18:39:23.745179 17738 heartbeater.cc:644] Failed to heartbeat to 127.0.198.62:38905 (0 consecutive failures): Network error: Failed to ping master at 127.0.198.62:38905: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
I20260929 18:39:23.745733 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.745531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:58559)
0929 18:39:23.745633 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.745680 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:39:23.746409 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:23.746212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:46103)
0929 18:39:23.746318 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:23.746360 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:39:23.746547 17738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:39:24.421815 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:24.421341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:35209)
0929 18:39:24.421632 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:24.421698 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260929 18:39:24.548754 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:24.548358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:56003)
0929 18:39:24.548594 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:24.548648 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260929 18:39:24.647433 17744 raft_consensus.cc:482] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:24.647675 17744 raft_consensus.cc:504] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:24.648828 17744 leader_election.cc:288] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715), 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317)
Sep 29 18:39:24 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.2@KRBTEST.COM
Sep 29 18:39:24 dist-test-slave-q5qx krb5kdc[16157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.3@KRBTEST.COM
I20260929 18:39:24.664979 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:24.648989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:33715 (local address 127.0.198.1:47123)
0929 18:39:24.649235 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:24.649250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:24.649355 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:24.651059 (+ 1704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:24.651065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:24.651576 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:24.652050 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:24.652064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.653321 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.653324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:24.654276 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:24.654328 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.654980 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.654984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:24.655143 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:24.655942 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:24.655957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:24.658457 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:24.661912 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:24.661916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:24.661918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:24.662522 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:24.662972 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:24.662975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:24.662977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:24.663108 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:24.663852 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:24.663859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:24.664220 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 18:39:24.664438 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 18:39:24.664774 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":116,"spinlock_wait_cycles":6784}
I20260929 18:39:24.666898 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:24.649300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:35427 (local address 127.0.198.2:38317)
0929 18:39:24.649593 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:24.649617 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:24.649650 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:39:24.650345 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:24.650351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:24.650698 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:24.650993 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:24.652210 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.653182 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.654399 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.654900 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.655103 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:24.658747 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:24.658757 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:24.658777 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:24.658819 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:24.661737 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:24.662513 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:24.662535 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:24.662541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:24.662702 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:24.663096 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:24.663100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:24.663102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:24.663694 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:24.663830 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:24.666214 (+ 2384us) server_negotiation.cc:299] Negotiation successful
0929 18:39:24.666705 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260929 18:39:24.667461 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:24.649072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:38317 (local address 127.0.198.1:35427)
0929 18:39:24.649448 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:24.649461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:24.649539 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:24.651059 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:24.651065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:24.651576 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:24.652049 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:24.652064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.653325 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.653328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:24.654291 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:24.654301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.655057 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.655060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:24.655147 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:24.655876 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:24.655898 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:24.658423 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:24.661880 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:24.661887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:24.661903 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:24.662393 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:24.662822 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:24.662825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:24.662827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:24.662997 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:24.665679 (+ 2682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:24.665682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:24.665781 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:39:24.665880 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:39:24.667296 (+ 1416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":173,"spinlock_wait_cycles":3968,"thread_start_us":78,"threads_started":1}
I20260929 18:39:24.671772 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:24.649204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:47123 (local address 127.0.198.3:33715)
0929 18:39:24.649590 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:24.649603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:24.649640 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:39:24.650363 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:24.650370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:24.650707 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:24.650964 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:24.652211 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.653189 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.654430 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:24.654869 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:24.655004 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:24.658849 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:24.658859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:24.658990 (+ 131us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:24.659060 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:24.661763 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:24.662658 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:24.662663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:24.662668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:24.662872 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:24.663209 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:24.663220 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:24.663222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:24.663694 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:24.663830 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:24.670379 (+ 6549us) server_negotiation.cc:299] Negotiation successful
0929 18:39:24.670702 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":65,"threads_started":1}
W20260929 18:39:24.671940 17357 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
W20260929 18:39:24.674322 17358 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
I20260929 18:39:24.674441 17358 leader_election.cc:302] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2da3c2d0cdea4bb6b16db539dc5a4905; no voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba
I20260929 18:39:24.674741 17744 raft_consensus.cc:2726] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:39:24.747529 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:24.747107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:60975)
0929 18:39:24.747358 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:24.747422 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260929 18:39:25.423305 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:25.422794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:38917)
0929 18:39:25.423123 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:25.423200 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
I20260929 18:39:25.550180 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:25.549647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:58873)
0929 18:39:25.549951 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:25.550081 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":80,"threads_started":1}
I20260929 18:39:25.748803 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:25.748341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:58063)
0929 18:39:25.748644 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:25.748709 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260929 18:39:26.424638 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:26.424182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:41495)
0929 18:39:26.424457 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:26.424520 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260929 18:39:26.551414 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:26.550978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:38489)
0929 18:39:26.551246 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:26.551304 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260929 18:39:26.750238 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:26.749677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:34073)
0929 18:39:26.750013 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:26.750076 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":77,"threads_started":1}
I20260929 18:39:26.778662 17755 raft_consensus.cc:482] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:26.778790 17755 raft_consensus.cc:504] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:26.779269 17755 leader_election.cc:288] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715), 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317)
W20260929 18:39:26.780247 17358 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
W20260929 18:39:26.780364 17357 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
I20260929 18:39:26.780452 17357 leader_election.cc:302] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2da3c2d0cdea4bb6b16db539dc5a4905; no voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba
I20260929 18:39:26.780596 17755 raft_consensus.cc:2726] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:39:27.425889 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:27.425446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:42063)
0929 18:39:27.425726 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:27.425790 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260929 18:39:27.552723 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:27.552226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:45799)
0929 18:39:27.552521 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:27.552603 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260929 18:39:27.751418 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:27.750980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:45459)
0929 18:39:27.751271 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:27.751324 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260929 18:39:28.427350 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:28.426842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:36753)
0929 18:39:28.427126 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:28.427187 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":73,"threads_started":1}
I20260929 18:39:28.554157 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:28.553594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:53445)
0929 18:39:28.553931 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:28.554027 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":77,"threads_started":1}
I20260929 18:39:28.752609 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:28.752146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:43381)
0929 18:39:28.752457 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:28.752516 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
I20260929 18:39:29.391042 17762 raft_consensus.cc:482] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:29.391185 17762 raft_consensus.cc:504] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:29.391691 17762 leader_election.cc:288] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715), 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317)
W20260929 18:39:29.392791 17358 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
W20260929 18:39:29.392940 17357 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
I20260929 18:39:29.393055 17357 leader_election.cc:302] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2da3c2d0cdea4bb6b16db539dc5a4905; no voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba
I20260929 18:39:29.393209 17762 raft_consensus.cc:2726] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:39:29.428694 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:29.428250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:44513)
0929 18:39:29.428535 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:29.428597 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260929 18:39:29.555480 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:29.554938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:46539)
0929 18:39:29.555314 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:29.555373 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":92,"threads_started":1}
I20260929 18:39:29.753719 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:29.753313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:32903)
0929 18:39:29.753548 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:29.753604 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260929 18:39:30.430236 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:30.429680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:44409)
0929 18:39:30.430058 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:30.430134 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":88,"threads_started":1}
I20260929 18:39:30.556787 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:30.556338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:59963)
0929 18:39:30.556628 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:30.556686 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":73,"threads_started":1}
I20260929 18:39:30.755082 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:30.754518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:54347)
0929 18:39:30.754892 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:30.754965 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":87,"threads_started":1}
I20260929 18:39:31.431582 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:31.431059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:45893)
0929 18:39:31.431420 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:31.431487 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":166,"threads_started":1}
I20260929 18:39:31.558123 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:31.557589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:42191)
0929 18:39:31.557918 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:31.558020 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":69,"threads_started":1}
I20260929 18:39:31.756379 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:31.755920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:43013)
0929 18:39:31.756213 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:31.756278 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260929 18:39:31.960319 17772 raft_consensus.cc:482] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:31.960465 17772 raft_consensus.cc:504] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:31.960929 17772 leader_election.cc:288] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715), 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317)
W20260929 18:39:31.961809 17357 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
W20260929 18:39:31.961932 17358 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
I20260929 18:39:31.962023 17358 leader_election.cc:302] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2da3c2d0cdea4bb6b16db539dc5a4905; no voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba
I20260929 18:39:31.962234 17772 raft_consensus.cc:2726] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:39:32.432945 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:32.432486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:37713)
0929 18:39:32.432784 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:32.432849 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20260929 18:39:32.559542 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:32.558997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:43377)
0929 18:39:32.559377 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:32.559445 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":154,"threads_started":1}
I20260929 18:39:32.757745 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:32.757267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:35881)
0929 18:39:32.757567 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:32.757635 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260929 18:39:33.434576 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:33.434109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:57547)
0929 18:39:33.434420 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:33.434477 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
I20260929 18:39:33.560815 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:33.560361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:41233)
0929 18:39:33.560639 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:33.560711 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260929 18:39:33.759097 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:33.758651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:50183)
0929 18:39:33.758939 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:33.759000 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260929 18:39:34.435803 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:34.435389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:41807)
0929 18:39:34.435647 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:34.435708 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
I20260929 18:39:34.562233 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:34.561639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:57989)
0929 18:39:34.561931 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:34.562041 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260929 18:39:34.760365 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:34.759903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:49169)
0929 18:39:34.760202 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:34.760267 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
I20260929 18:39:35.437028 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:35.436596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:52679)
0929 18:39:35.436874 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:35.436932 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260929 18:39:35.563495 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:35.562980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:53945)
0929 18:39:35.563326 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:35.563390 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
I20260929 18:39:35.761577 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:35.761152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:45203)
0929 18:39:35.761423 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:35.761483 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260929 18:39:36.438416 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:36.437863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:51479)
0929 18:39:36.438252 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:36.438316 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260929 18:39:36.564797 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:36.564360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:52107)
0929 18:39:36.564632 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:36.564700 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260929 18:39:36.762902 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:36.762444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:40629)
0929 18:39:36.762743 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:36.762808 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260929 18:39:36.917537 17788 raft_consensus.cc:482] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:36.917686 17788 raft_consensus.cc:504] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a750095c0ccd49de8ef4f2d7dc8901ba" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 33715 } } peers { permanent_uuid: "5b7738d7ac7d4ad286863268d3f0ff25" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 38317 } } peers { permanent_uuid: "2da3c2d0cdea4bb6b16db539dc5a4905" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 35053 } }
I20260929 18:39:36.918236 17788 leader_election.cc:288] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715), 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317)
W20260929 18:39:36.919128 17358 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a750095c0ccd49de8ef4f2d7dc8901ba (127.0.198.3:33715): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
W20260929 18:39:36.919243 17357 leader_election.cc:341] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b7738d7ac7d4ad286863268d3f0ff25 (127.0.198.2:38317): Not found: Tablet not found: 06cb40df419f404ea38a558f88d48b34
I20260929 18:39:36.919333 17357 leader_election.cc:302] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2da3c2d0cdea4bb6b16db539dc5a4905; no voters: 5b7738d7ac7d4ad286863268d3f0ff25, a750095c0ccd49de8ef4f2d7dc8901ba
I20260929 18:39:36.919492 17788 raft_consensus.cc:2726] T 06cb40df419f404ea38a558f88d48b34 P 2da3c2d0cdea4bb6b16db539dc5a4905 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:39:37.439996 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:37.439215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:47171)
0929 18:39:37.439798 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:37.439874 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":77,"threads_started":1}
I20260929 18:39:37.566177 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:37.565618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:42217)
0929 18:39:37.565891 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:37.565952 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260929 18:39:37.764247 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:37.763736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.3:51753)
0929 18:39:37.764043 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:37.764105 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
I20260929 18:39:38.441596 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:38.441161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.1:57323)
0929 18:39:38.441439 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:38.441497 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260929 18:39:38.567656 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:38.567148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:38905 (local address 127.0.198.2:35873)
0929 18:39:38.567451 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:38.567522 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.62:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260929 18:39:38.744256 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 17301
I20260929 18:39:38.750121 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 17473
I20260929 18:39:38.754897 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 17608
2026-09-29T18:39:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59296 ms)
[----------] 30 tests from SecurityITest (236001 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:38 dist-test-slave-q5qx krb5kdc[17796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:38 dist-test-slave-q5qx krb5kdc[17796](info): set up 2 sockets
Sep 29 18:39:38 dist-test-slave-q5qx krb5kdc[17796](info): commencing operation
krb5kdc: starting...
W20260929 18:39:39.721735 792 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:39 dist-test-slave-q5qx krb5kdc[17796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:39:39.846504 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41239
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:39.956529 17809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:39.956812 17809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:39.956869 17809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:39.960927 17809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:39.960999 17809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:39.961025 17809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:39.961045 17809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:39.961066 17809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:39.966480 17809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:39.967566 17809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:39.968619 17809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:39.977322 17814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:39.977331 17815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:39.977447 17817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:39.977478 17809 server_base.cc:1034] running on GCE node
I20260929 18:39:39.978246 17809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:39.979336 17809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:39.980506 17809 hybrid_clock.cc:648] HybridClock initialized: now 1790707179980473 us; error 45 us; skew 500 ppm
Sep 29 18:39:39 dist-test-slave-q5qx krb5kdc[17796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:39.983513 17809 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:39:39.984416 17809 webserver.cc:460] Webserver started at http://127.0.198.62:46833/ using document root <none> and password file <none>
I20260929 18:39:39.984931 17809 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:39.984992 17809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:39.985157 17809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:39.987208 17809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "3e3bfc454fd04fad9e911c978ee5a485"
format_stamp: "Formatted at 2026-09-29 18:39:39 on dist-test-slave-q5qx"
server_key: "d08c744a098415e79a521a1b6ceb6ba0"
server_key_iv: "9738369d7c1aa06479414a0fac79773b"
server_key_version: "encryptionkey@0"
I20260929 18:39:39.987712 17809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "3e3bfc454fd04fad9e911c978ee5a485"
format_stamp: "Formatted at 2026-09-29 18:39:39 on dist-test-slave-q5qx"
server_key: "d08c744a098415e79a521a1b6ceb6ba0"
server_key_iv: "9738369d7c1aa06479414a0fac79773b"
server_key_version: "encryptionkey@0"
I20260929 18:39:39.991739 17809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:39:39.994481 17824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:39.995982 17809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:39.996181 17809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "3e3bfc454fd04fad9e911c978ee5a485"
format_stamp: "Formatted at 2026-09-29 18:39:39 on dist-test-slave-q5qx"
server_key: "d08c744a098415e79a521a1b6ceb6ba0"
server_key_iv: "9738369d7c1aa06479414a0fac79773b"
server_key_version: "encryptionkey@0"
I20260929 18:39:39.996302 17809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:40.021082 17809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:40.024530 17809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:40.024731 17809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:40.031800 17809 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:41239
I20260929 18:39:40.031817 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:41239 every 8 connection(s)
I20260929 18:39:40.032785 17809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:40.033082 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 17809
I20260929 18:39:40.033202 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:40.033519 792 external_mini_cluster.cc:1428] Setting key faa65e6023ae3fcdb078303146c1418a
I20260929 18:39:40.035867 17877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:40.041721 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485: Bootstrap starting.
Sep 29 18:39:40 dist-test-slave-q5qx krb5kdc[17796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:40.044353 17877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:40.045274 17877 log.cc:824] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:40.047821 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485: No bootstrap required, opened a new log
I20260929 18:39:40.048883 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.034955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.0.198.62:41239)
0929 18:39:40.035408 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.035423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.035458 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:39:40.036168 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.036187 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.036521 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.036854 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:40.037711 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.038780 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.039696 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.040385 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.040547 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.043339 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:40.043345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:40.043361 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:40.043396 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:40.045366 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.045990 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.045997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.046003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.046230 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.046541 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.046544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.046546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.047130 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:40.047332 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.047603 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.047910 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
I20260929 18:39:40.051916 17877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bfc454fd04fad9e911c978ee5a485" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41239 } }
I20260929 18:39:40.052153 17877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:40.052189 17877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3bfc454fd04fad9e911c978ee5a485, State: Initialized, Role: FOLLOWER
I20260929 18:39:40.052552 17877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bfc454fd04fad9e911c978ee5a485" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41239 } }
I20260929 18:39:40.052698 17877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:40.052767 17877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:40.052868 17877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:40.053768 17877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bfc454fd04fad9e911c978ee5a485" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41239 } }
I20260929 18:39:40.054121 17877 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e3bfc454fd04fad9e911c978ee5a485; no voters:
I20260929 18:39:40.054410 17877 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:40.054514 17882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:40.054768 17882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [term 1 LEADER]: Becoming Leader. State: Replica: 3e3bfc454fd04fad9e911c978ee5a485, State: Running, Role: LEADER
I20260929 18:39:40.055054 17882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bfc454fd04fad9e911c978ee5a485" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41239 } }
I20260929 18:39:40.055557 17877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:40.056562 17884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3bfc454fd04fad9e911c978ee5a485. Latest consensus state: current_term: 1 leader_uuid: "3e3bfc454fd04fad9e911c978ee5a485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bfc454fd04fad9e911c978ee5a485" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41239 } } }
I20260929 18:39:40.056708 17884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:40.057001 17883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e3bfc454fd04fad9e911c978ee5a485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bfc454fd04fad9e911c978ee5a485" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 41239 } } }
I20260929 18:39:40.057090 17891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:40.057126 17883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:40.060072 17891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:40.066303 17891 catalog_manager.cc:1347] Generated new cluster ID: cbbf8f904295409ca3ad8d527926f11f
I20260929 18:39:40.066398 17891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:40.083496 17891 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:40.084897 17891 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:40.096226 17891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485: Generated new TSK 0
I20260929 18:39:40.097121 17891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:39:40.129856 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41239
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:40.241146 17903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:40.241433 17903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:40.241475 17903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:40.245579 17903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:40.245640 17903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:40.245703 17903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:40.251034 17903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:40.252053 17903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:40.253050 17903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:40.262661 17908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:40.262642 17911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:40.262639 17909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:40.262956 17903 server_base.cc:1034] running on GCE node
I20260929 18:39:40.263356 17903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:40.263962 17903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:40.265152 17903 hybrid_clock.cc:648] HybridClock initialized: now 1790707180265119 us; error 50 us; skew 500 ppm
Sep 29 18:39:40 dist-test-slave-q5qx krb5kdc[17796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:40.267931 17903 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:39:40.268824 17903 webserver.cc:460] Webserver started at http://127.0.198.1:42431/ using document root <none> and password file <none>
I20260929 18:39:40.269357 17903 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:40.269423 17903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:40.269618 17903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:40.271663 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "4191bed86f1541dfa10adb843ecf3f74"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "11ed3b98777a1ec5b02b2199841de1cb"
server_key_iv: "9d7667c3f57831bb81ba39d6c09e9c45"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.272202 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "4191bed86f1541dfa10adb843ecf3f74"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "11ed3b98777a1ec5b02b2199841de1cb"
server_key_iv: "9d7667c3f57831bb81ba39d6c09e9c45"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.276113 17903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 18:39:40.278718 17918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.280025 17903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:40.280237 17903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "4191bed86f1541dfa10adb843ecf3f74"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "11ed3b98777a1ec5b02b2199841de1cb"
server_key_iv: "9d7667c3f57831bb81ba39d6c09e9c45"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.280344 17903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:40.295181 17903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:40.298460 17903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:40.298643 17903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:40.299227 17903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:40.300098 17903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:40.300143 17903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.300196 17903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:40.300225 17903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.308722 17903 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:38747
I20260929 18:39:40.308755 18031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:38747 every 8 connection(s)
I20260929 18:39:40.309595 17903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:40.316274 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 17903
I20260929 18:39:40.316426 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:40.316689 792 external_mini_cluster.cc:1428] Setting key 3bc711b25d5034ef9a010bb3ae37cbe1
Sep 29 18:39:40 dist-test-slave-q5qx krb5kdc[17796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:40.324361 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.311498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:57313 (local address 127.0.198.62:41239)
0929 18:39:40.311698 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.311702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.312540 (+ 838us) server_negotiation.cc:407] Connection header received
0929 18:39:40.313888 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.313891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.313951 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.314046 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:40.315566 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.316199 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.317603 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.317932 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.318084 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.320846 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:40.320850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:40.320853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:40.320879 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:40.322309 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.322813 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.322817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.322819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.322897 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.323233 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.323236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.323238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.323442 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:40.323520 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.324055 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.324192 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 18:39:40.325291 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.311877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41239 (local address 127.0.198.1:57313)
0929 18:39:40.312375 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.312423 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.313633 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.314228 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.314249 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.314775 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:40.315409 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.315424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.316369 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.316372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.317479 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.317490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.318128 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.318132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.318435 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.319054 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:40.319075 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:40.320671 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:40.322423 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:40.322429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:40.322448 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:40.322714 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:40.322987 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:40.322990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:40.322992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:40.323145 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:40.323550 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:40.323558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:40.323820 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.323998 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.324361 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:39:40.326598 18032 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41239
I20260929 18:39:40.326886 18032 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:40.327483 18032 heartbeater.cc:511] Master 127.0.198.62:41239 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:39:40.329149 17841 ts_manager.cc:194] Registered new tserver with Master: 4191bed86f1541dfa10adb843ecf3f74 (127.0.198.1:38747)
I20260929 18:39:40.330338 17841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:57313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:39:40.345713 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41239
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:40.468783 18037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:40.469066 18037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:40.469110 18037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:40.473235 18037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:40.473316 18037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:40.473379 18037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:40.478873 18037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:40.479929 18037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:40.480931 18037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:40.491698 18043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:40.491645 18042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:40.491657 18045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:40.491746 18037 server_base.cc:1034] running on GCE node
I20260929 18:39:40.492744 18037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:40.493502 18037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:40.494700 18037 hybrid_clock.cc:648] HybridClock initialized: now 1790707180494680 us; error 38 us; skew 500 ppm
Sep 29 18:39:40 dist-test-slave-q5qx krb5kdc[17796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:40.497941 18037 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:39:40.498966 18037 webserver.cc:460] Webserver started at http://127.0.198.2:41459/ using document root <none> and password file <none>
I20260929 18:39:40.499647 18037 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:40.499723 18037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:40.499927 18037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:40.502182 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "9305a1653f2445e5b2931b0e84a59d88"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "1c36b78b0fe89ba61617df1c980298ba"
server_key_iv: "44bcfcf37473125777ce05b4feb34b0f"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.502756 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "9305a1653f2445e5b2931b0e84a59d88"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "1c36b78b0fe89ba61617df1c980298ba"
server_key_iv: "44bcfcf37473125777ce05b4feb34b0f"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.506660 18037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:39:40.509250 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.510586 18037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:40.510727 18037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "9305a1653f2445e5b2931b0e84a59d88"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "1c36b78b0fe89ba61617df1c980298ba"
server_key_iv: "44bcfcf37473125777ce05b4feb34b0f"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.510834 18037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:40.528759 18037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:40.531975 18037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:40.532152 18037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:40.532745 18037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:40.533583 18037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:40.533628 18037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.533690 18037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:40.533715 18037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.542428 18037 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42015
I20260929 18:39:40.542501 18165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42015 every 8 connection(s)
I20260929 18:39:40.543347 18037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:40.553217 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18037
I20260929 18:39:40.553336 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
Sep 29 18:39:40 dist-test-slave-q5qx krb5kdc[17796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:40.553644 792 external_mini_cluster.cc:1428] Setting key 361c9da125c2b18c3c3df536b228b290
I20260929 18:39:40.558290 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.545314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:36971 (local address 127.0.198.62:41239)
0929 18:39:40.545489 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.545493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.546422 (+ 929us) server_negotiation.cc:407] Connection header received
0929 18:39:40.547694 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.547697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.547746 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.547839 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:40.549250 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.549826 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.551210 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.551449 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.551544 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.554352 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:40.554356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:40.554359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:40.554389 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:40.555888 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.556502 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.556505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.556506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.556659 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.556986 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.556989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.556991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.557177 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:40.557262 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.558002 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.558129 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 18:39:40.559147 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.545691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41239 (local address 127.0.198.2:36971)
0929 18:39:40.546276 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.546321 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.547480 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.548028 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.548038 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.548515 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:40.549014 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.549028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.550007 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.550012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.551067 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.551079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.551602 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.551606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.551762 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.552382 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:40.552399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:40.554106 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:40.556015 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:40.556020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:40.556033 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:40.556361 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:40.556740 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:40.556744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:40.556746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:40.556890 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:40.557272 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:40.557279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:40.557563 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.557899 (+ 336us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.558293 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":116,"threads_started":1}
I20260929 18:39:40.560277 18166 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41239
I20260929 18:39:40.560526 18166 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:40.561036 18166 heartbeater.cc:511] Master 127.0.198.62:41239 requested a full tablet report, sending...
I20260929 18:39:40.562125 17841 ts_manager.cc:194] Registered new tserver with Master: 9305a1653f2445e5b2931b0e84a59d88 (127.0.198.2:42015)
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:39:40.562899 17841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:36971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:39:40.581959 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41239
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:40.692103 18171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:40.692387 18171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:40.692430 18171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:40.696547 18171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:40.696610 18171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:40.696671 18171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:40.702158 18171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:41239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:40.703183 18171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:40.704164 18171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:40.714087 18177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:40.714174 18176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:40.714147 18171 server_base.cc:1034] running on GCE node
W20260929 18:39:40.714087 18179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:40.714716 18171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:40.715440 18171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:40.716595 18171 hybrid_clock.cc:648] HybridClock initialized: now 1790707180716592 us; error 40 us; skew 500 ppm
Sep 29 18:39:40 dist-test-slave-q5qx krb5kdc[17796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:40.719506 18171 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:39:40.720417 18171 webserver.cc:460] Webserver started at http://127.0.198.3:40419/ using document root <none> and password file <none>
I20260929 18:39:40.720968 18171 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:40.721035 18171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:40.721243 18171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:40.723443 18171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "058bd93ff62d49feb22ea5b36ec3778c"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "ade82060a617e2c1e5602ef212d7e1f9"
server_key_iv: "fc459dee5b8de809735b3b6e05b9761d"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.724010 18171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "058bd93ff62d49feb22ea5b36ec3778c"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "ade82060a617e2c1e5602ef212d7e1f9"
server_key_iv: "fc459dee5b8de809735b3b6e05b9761d"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.727969 18171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:39:40.730674 18186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.732141 18171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:40.732277 18171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "058bd93ff62d49feb22ea5b36ec3778c"
format_stamp: "Formatted at 2026-09-29 18:39:40 on dist-test-slave-q5qx"
server_key: "ade82060a617e2c1e5602ef212d7e1f9"
server_key_iv: "fc459dee5b8de809735b3b6e05b9761d"
server_key_version: "encryptionkey@0"
I20260929 18:39:40.732385 18171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:40.776260 18171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:40.779600 18171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:40.779789 18171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:40.780382 18171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:40.781206 18171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:40.781251 18171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.781303 18171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:40.781333 18171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:40.790357 18171 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:46317
I20260929 18:39:40.790378 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:46317 every 8 connection(s)
I20260929 18:39:40.791232 18171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:40.799090 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18171
I20260929 18:39:40.799247 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:40.799562 792 external_mini_cluster.cc:1428] Setting key 87c20a4a8c3dc8ebcf4a04d838fdcbd3
Sep 29 18:39:40 dist-test-slave-q5qx krb5kdc[17796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:40.804618 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.793090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:58397 (local address 127.0.198.62:41239)
0929 18:39:40.793258 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.793263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.794043 (+ 780us) server_negotiation.cc:407] Connection header received
0929 18:39:40.794999 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.795002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.795048 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.795124 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:40.796423 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.796965 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.798338 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.798560 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.798642 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.801360 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:40.801365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:40.801367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:40.801396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:40.802680 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.803164 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.803167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.803168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.803240 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.803618 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.803621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.803622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.803795 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:40.803872 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.804378 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.804476 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:39:40.805423 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.793426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:41239 (local address 127.0.198.3:58397)
0929 18:39:40.793887 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.793930 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.794803 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.795335 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.795342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.795728 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:40.796258 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.796273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.797180 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.797183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.798224 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.798234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.798701 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.798704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.798893 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.799579 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:40.799601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:40.801204 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:40.802777 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:40.802783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:40.802796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:40.803070 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:40.803342 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:40.803344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:40.803345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:40.803493 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:40.803873 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:40.803878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:40.804137 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.804319 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.804619 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260929 18:39:40.806553 18300 heartbeater.cc:347] Connected to a master server at 127.0.198.62:41239
I20260929 18:39:40.806761 18300 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:40.807261 18300 heartbeater.cc:511] Master 127.0.198.62:41239 requested a full tablet report, sending...
I20260929 18:39:40.808231 17841 ts_manager.cc:194] Registered new tserver with Master: 058bd93ff62d49feb22ea5b36ec3778c (127.0.198.3:46317)
I20260929 18:39:40.808805 17841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:58397
I20260929 18:39:40.813549 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:40.823081 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.815621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.0.198.62:41239)
0929 18:39:40.815744 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.815749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.815941 (+ 192us) server_negotiation.cc:407] Connection header received
0929 18:39:40.816032 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.816035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.816078 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.816141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:40.816863 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.817452 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.818474 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.818700 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.818799 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.820078 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:40.820081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:40.820083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:40.820106 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:40.821456 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.821880 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.821882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.821884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.821949 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:40.822298 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:40.822300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:40.822301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:40.822515 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:40.822614 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.822829 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.822925 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:39:40.825752 17841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:40.828850 17841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:40.842907 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.838171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.0.198.2:42015)
0929 18:39:40.838490 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.838502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.838524 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:39:40.838598 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.838603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.838800 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.839194 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:40.839317 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.840192 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.841438 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.842227 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.842330 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.842467 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.842653 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.842745 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260929 18:39:40.842906 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.837292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:46317 (local address 127.0.0.1:59250)
0929 18:39:40.837709 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.837755 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.837874 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.838624 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.838631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.838654 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:40.838781 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.838788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.840091 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.840094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.841204 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.841215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.842538 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.842541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.842618 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.842636 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.842690 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.842743 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":71,"threads_started":1}
I20260929 18:39:40.842906 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.837422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42015 (local address 127.0.0.1:57130)
0929 18:39:40.838106 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.838119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.838196 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.838937 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.838939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.838954 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:40.839056 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.839063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.840325 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.840329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.841306 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.841317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.842350 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.842354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.842466 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.842503 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.842609 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.842744 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":8,"thread_start_us":95,"threads_started":1}
I20260929 18:39:40.843498 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.837415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.0.198.3:46317)
0929 18:39:40.837887 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.837900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.837924 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:39:40.838065 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.838071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.838284 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.838436 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:40.838903 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.839975 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.841463 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.842257 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.842348 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.842482 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.843307 (+ 825us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.843387 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":169,"threads_started":1}
I20260929 18:39:40.843590 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.838023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38747 (local address 127.0.0.1:53978)
0929 18:39:40.838792 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.838804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.838882 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.839555 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.839558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.839577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:40.839658 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.839664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.840851 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.840854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.841922 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.841932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.843221 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.843224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.843313 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.843335 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.843394 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.843464 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260929 18:39:40.844000 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.838438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.0.198.1:38747)
0929 18:39:40.839163 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.839169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.839190 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:39:40.839258 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.839262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.839423 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.839552 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:40.839762 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.840731 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.842061 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.842760 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.842854 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.843709 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.843779 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.843866 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":104,"threads_started":1}
I20260929 18:39:40.846071 18100 tablet_service.cc:1467] Processing CreateTablet for tablet 81acff323d6d4e5891b4f71561d5408f (DEFAULT_TABLE table=test-table [id=e08065752f314da7a7d34634aed4819b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:40.846237 18234 tablet_service.cc:1467] Processing CreateTablet for tablet 81acff323d6d4e5891b4f71561d5408f (DEFAULT_TABLE table=test-table [id=e08065752f314da7a7d34634aed4819b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:40.847011 18100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81acff323d6d4e5891b4f71561d5408f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:40.847151 18234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81acff323d6d4e5891b4f71561d5408f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:40.848469 17966 tablet_service.cc:1467] Processing CreateTablet for tablet 81acff323d6d4e5891b4f71561d5408f (DEFAULT_TABLE table=test-table [id=e08065752f314da7a7d34634aed4819b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:40.849437 17966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81acff323d6d4e5891b4f71561d5408f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:40.854063 18317 tablet_bootstrap.cc:492] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Bootstrap starting.
I20260929 18:39:40.854334 18318 tablet_bootstrap.cc:492] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Bootstrap starting.
I20260929 18:39:40.855959 18319 tablet_bootstrap.cc:492] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74: Bootstrap starting.
I20260929 18:39:40.856530 18318 tablet_bootstrap.cc:654] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:40.857017 18317 tablet_bootstrap.cc:654] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:40.857746 18317 log.cc:824] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:40.857746 18318 log.cc:824] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:40.857950 18319 tablet_bootstrap.cc:654] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:40.858734 18319 log.cc:824] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:40.859797 18318 tablet_bootstrap.cc:492] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: No bootstrap required, opened a new log
I20260929 18:39:40.859795 18317 tablet_bootstrap.cc:492] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: No bootstrap required, opened a new log
I20260929 18:39:40.859984 18318 ts_tablet_manager.cc:1397] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:39:40.859984 18317 ts_tablet_manager.cc:1397] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:39:40.860431 18319 tablet_bootstrap.cc:492] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74: No bootstrap required, opened a new log
I20260929 18:39:40.860605 18319 ts_tablet_manager.cc:1397] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260929 18:39:40.863840 18319 raft_consensus.cc:348] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.864055 18319 raft_consensus.cc:374] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:40.863867 18318 raft_consensus.cc:348] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.863867 18317 raft_consensus.cc:348] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.864100 18319 raft_consensus.cc:729] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4191bed86f1541dfa10adb843ecf3f74, State: Initialized, Role: FOLLOWER
I20260929 18:39:40.864117 18317 raft_consensus.cc:374] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:40.864117 18318 raft_consensus.cc:374] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:40.864163 18317 raft_consensus.cc:729] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9305a1653f2445e5b2931b0e84a59d88, State: Initialized, Role: FOLLOWER
I20260929 18:39:40.864163 18318 raft_consensus.cc:729] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058bd93ff62d49feb22ea5b36ec3778c, State: Initialized, Role: FOLLOWER
I20260929 18:39:40.864580 18319 consensus_queue.cc:260] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.864589 18318 consensus_queue.cc:260] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.864589 18317 consensus_queue.cc:260] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.865206 18032 heartbeater.cc:503] Master 127.0.198.62:41239 was elected leader, sending a full tablet report...
I20260929 18:39:40.865473 18319 ts_tablet_manager.cc:1428] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 18:39:40.865780 18317 ts_tablet_manager.cc:1428] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:39:40.865958 18166 heartbeater.cc:503] Master 127.0.198.62:41239 was elected leader, sending a full tablet report...
I20260929 18:39:40.866115 18324 raft_consensus.cc:482] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:40.866246 18318 ts_tablet_manager.cc:1428] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 18:39:40.866241 18324 raft_consensus.cc:504] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.866551 18300 heartbeater.cc:503] Master 127.0.198.62:41239 was elected leader, sending a full tablet report...
I20260929 18:39:40.868618 18324 leader_election.cc:288] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 058bd93ff62d49feb22ea5b36ec3778c (127.0.198.3:46317), 4191bed86f1541dfa10adb843ecf3f74 (127.0.198.1:38747)
I20260929 18:39:40.872321 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.868207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:45947 (local address 127.0.198.3:46317)
0929 18:39:40.868302 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.868307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.868320 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:39:40.868477 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.868481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.868522 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.868582 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:40.868944 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.869553 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.871048 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.871656 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.871762 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.871823 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.872164 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.872229 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:39:40.872432 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.868094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:46317 (local address 127.0.198.2:45947)
0929 18:39:40.868268 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.868282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.868362 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.868700 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.868703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.868720 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:40.868831 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.868838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.869678 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.869681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.870923 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.870933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.871877 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.871880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.871999 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.872021 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.872126 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.872274 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":18}
I20260929 18:39:40.873361 18254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acff323d6d4e5891b4f71561d5408f" candidate_uuid: "9305a1653f2445e5b2931b0e84a59d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058bd93ff62d49feb22ea5b36ec3778c" is_pre_election: true
I20260929 18:39:40.873605 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.868910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38747 (local address 127.0.198.2:48733)
0929 18:39:40.869275 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.869288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:40.869380 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:40.870102 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:40.870105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:40.870123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:40.870233 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.870239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.871135 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.871138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.872293 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:40.872302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.873261 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.873264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:40.873336 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.873355 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:39:40.873422 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:39:40.873472 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":28,"thread_start_us":89,"threads_started":1}
I20260929 18:39:40.873644 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.869016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:48733 (local address 127.0.198.1:38747)
0929 18:39:40.869121 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.869126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.869374 (+ 248us) server_negotiation.cc:407] Connection header received
0929 18:39:40.869501 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.869504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.869546 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.869618 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:40.870349 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.871022 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.872425 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.873043 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.873162 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.873220 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.873455 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.873539 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:39:40.873647 18254 raft_consensus.cc:2454] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9305a1653f2445e5b2931b0e84a59d88 in term 0.
I20260929 18:39:40.874136 18056 leader_election.cc:302] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bd93ff62d49feb22ea5b36ec3778c, 9305a1653f2445e5b2931b0e84a59d88; no voters:
I20260929 18:39:40.874421 18324 raft_consensus.cc:2781] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:40.874437 17986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acff323d6d4e5891b4f71561d5408f" candidate_uuid: "9305a1653f2445e5b2931b0e84a59d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4191bed86f1541dfa10adb843ecf3f74" is_pre_election: true
I20260929 18:39:40.874504 18324 raft_consensus.cc:482] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:40.874557 18324 raft_consensus.cc:3037] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:40.874667 17986 raft_consensus.cc:2454] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9305a1653f2445e5b2931b0e84a59d88 in term 0.
I20260929 18:39:40.875710 18324 raft_consensus.cc:504] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.876041 18324 leader_election.cc:288] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [CANDIDATE]: Term 1 election: Requested vote from peers 058bd93ff62d49feb22ea5b36ec3778c (127.0.198.3:46317), 4191bed86f1541dfa10adb843ecf3f74 (127.0.198.1:38747)
I20260929 18:39:40.876338 18254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acff323d6d4e5891b4f71561d5408f" candidate_uuid: "9305a1653f2445e5b2931b0e84a59d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058bd93ff62d49feb22ea5b36ec3778c"
I20260929 18:39:40.876405 17986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acff323d6d4e5891b4f71561d5408f" candidate_uuid: "9305a1653f2445e5b2931b0e84a59d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4191bed86f1541dfa10adb843ecf3f74"
I20260929 18:39:40.876441 18254 raft_consensus.cc:3037] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:40.876526 17986 raft_consensus.cc:3037] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:40.877439 18254 raft_consensus.cc:2454] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9305a1653f2445e5b2931b0e84a59d88 in term 1.
I20260929 18:39:40.877525 17986 raft_consensus.cc:2454] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9305a1653f2445e5b2931b0e84a59d88 in term 1.
I20260929 18:39:40.877720 18056 leader_election.cc:302] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bd93ff62d49feb22ea5b36ec3778c, 9305a1653f2445e5b2931b0e84a59d88; no voters:
I20260929 18:39:40.877899 18324 raft_consensus.cc:2781] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:40.878014 18324 raft_consensus.cc:686] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 1 LEADER]: Becoming Leader. State: Replica: 9305a1653f2445e5b2931b0e84a59d88, State: Running, Role: LEADER
I20260929 18:39:40.878329 18324 consensus_queue.cc:237] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } }
I20260929 18:39:40.881426 17841 catalog_manager.cc:5495] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9305a1653f2445e5b2931b0e84a59d88 (127.0.198.2). New cstate: current_term: 1 leader_uuid: "9305a1653f2445e5b2931b0e84a59d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9305a1653f2445e5b2931b0e84a59d88" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 } health_report { overall_health: UNKNOWN } } }
I20260929 18:39:40.893353 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.889891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.198.2:42015)
0929 18:39:40.890028 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:40.890032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:40.890044 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:39:40.890212 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:40.890215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:40.890267 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:40.890359 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:40.890555 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.891034 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.891880 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:40.892092 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:40.892182 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:40.892540 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:40.892964 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:40.893076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:40.893193 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 18:39:40.893245 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:39:40.899427 17986 raft_consensus.cc:1261] T 81acff323d6d4e5891b4f71561d5408f P 4191bed86f1541dfa10adb843ecf3f74 [term 1 FOLLOWER]: Refusing update from remote peer 9305a1653f2445e5b2931b0e84a59d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:40.899448 18254 raft_consensus.cc:1261] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 1 FOLLOWER]: Refusing update from remote peer 9305a1653f2445e5b2931b0e84a59d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:40.900309 18324 consensus_queue.cc:1035] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "058bd93ff62d49feb22ea5b36ec3778c" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:40.900530 18326 consensus_queue.cc:1035] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4191bed86f1541dfa10adb843ecf3f74" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:40.909160 18332 mvcc.cc:205] Tried to move back new op lower bound from 7334736612956172288 to 7334736612880105472. Current Snapshot: MvccSnapshot[applied={T|T < 7334736612956172288}]
I20260929 18:39:40.909433 18331 mvcc.cc:205] Tried to move back new op lower bound from 7334736612956172288 to 7334736612880105472. Current Snapshot: MvccSnapshot[applied={T|T < 7334736612956172288}]
I20260929 18:39:40.917042 17839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:40.917253 17839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:40.919806 17839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e3bfc454fd04fad9e911c978ee5a485: Sending DeleteTablet for 3 replicas of tablet 81acff323d6d4e5891b4f71561d5408f
I20260929 18:39:40.920589 18100 tablet_service.cc:1514] Processing DeleteTablet for tablet 81acff323d6d4e5891b4f71561d5408f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:40 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:57130
I20260929 18:39:40.920672 18234 tablet_service.cc:1514] Processing DeleteTablet for tablet 81acff323d6d4e5891b4f71561d5408f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:40 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:59250
I20260929 18:39:40.921088 17966 tablet_service.cc:1514] Processing DeleteTablet for tablet 81acff323d6d4e5891b4f71561d5408f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:40 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:53978
I20260929 18:39:40.921425 18339 tablet_replica.cc:330] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: stopping tablet replica
I20260929 18:39:40.921602 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 17903
I20260929 18:39:40.921762 18339 raft_consensus.cc:2229] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:39:40.922254 18339 raft_consensus.cc:2258] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:40.923033 18340 tablet_replica.cc:330] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: stopping tablet replica
I20260929 18:39:40.923357 18340 raft_consensus.cc:2229] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:40.923660 18340 raft_consensus.cc:2258] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:40.924350 18339 ts_tablet_manager.cc:1905] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:40.926990 18339 ts_tablet_manager.cc:1918] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:40.927067 18339 log.cc:1196] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/81acff323d6d4e5891b4f71561d5408f
I20260929 18:39:40.927446 18339 ts_tablet_manager.cc:1939] T 81acff323d6d4e5891b4f71561d5408f P 9305a1653f2445e5b2931b0e84a59d88: Deleting consensus metadata
I20260929 18:39:40.928507 17825 catalog_manager.cc:4841] TS 9305a1653f2445e5b2931b0e84a59d88 (127.0.198.2:42015): tablet 81acff323d6d4e5891b4f71561d5408f (table test-table [id=e08065752f314da7a7d34634aed4819b]) successfully deleted
I20260929 18:39:40.931257 18340 ts_tablet_manager.cc:1905] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:40.933295 18340 ts_tablet_manager.cc:1918] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:40.933377 18340 log.cc:1196] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/81acff323d6d4e5891b4f71561d5408f
I20260929 18:39:40.933627 18340 ts_tablet_manager.cc:1939] T 81acff323d6d4e5891b4f71561d5408f P 058bd93ff62d49feb22ea5b36ec3778c: Deleting consensus metadata
I20260929 18:39:40.935016 17828 catalog_manager.cc:4841] TS 058bd93ff62d49feb22ea5b36ec3778c (127.0.198.3:46317): tablet 81acff323d6d4e5891b4f71561d5408f (table test-table [id=e08065752f314da7a7d34634aed4819b]) successfully deleted
W20260929 18:39:40.935436 17828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:38747 (error 108)
I20260929 18:39:40.935743 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18037
I20260929 18:39:40.937690 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:40.937430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38747 (local address 127.0.0.1:53994)
0929 18:39:40.937548 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:40.937614 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 18:39:40.937851 17828 catalog_manager.cc:4584] TS 4191bed86f1541dfa10adb843ecf3f74 (127.0.198.1:38747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81acff323d6d4e5891b4f71561d5408f: Network error: Client connection negotiation failed: client connection to 127.0.198.1:38747: connect: Connection refused (error 111)
I20260929 18:39:40.941252 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18171
I20260929 18:39:40.946128 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 17809
2026-09-29T18:39:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2191 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T18:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:40Z Disabled control of system clock
I20260929 18:39:40.974013 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39093
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:39:41.086773 18346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:41.087076 18346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:41.087113 18346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:41.091167 18346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:41.091230 18346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:41.091244 18346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:41.091260 18346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:41.091275 18346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:41.096158 18346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:39093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:41.097236 18346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:41.098627 18346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:39:41.107120 18346 server_base.cc:1034] running on GCE node
W20260929 18:39:41.107112 18354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:41.107174 18352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:41.107191 18351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:41.107982 18346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:41.109143 18346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:41.110404 18346 hybrid_clock.cc:648] HybridClock initialized: now 1790707181110380 us; error 37 us; skew 500 ppm
I20260929 18:39:41.112378 18346 webserver.cc:460] Webserver started at http://127.0.198.62:34253/ using document root <none> and password file <none>
I20260929 18:39:41.112933 18346 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:41.112993 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:41.113169 18346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:41.115245 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "289f4122b4774b83ae698d4dff91c60f"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "d01bf024f44a3da5f114f3db12ad0404"
server_key_iv: "1a9be618970492572074ace998516d5e"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.115751 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "289f4122b4774b83ae698d4dff91c60f"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "d01bf024f44a3da5f114f3db12ad0404"
server_key_iv: "1a9be618970492572074ace998516d5e"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.119493 18346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:39:41.121934 18360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.123248 18346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:41.123351 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "289f4122b4774b83ae698d4dff91c60f"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "d01bf024f44a3da5f114f3db12ad0404"
server_key_iv: "1a9be618970492572074ace998516d5e"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.123440 18346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:41.136754 18346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:41.137436 18346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:41.137588 18346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:41.144183 18346 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:39093
I20260929 18:39:41.144217 18412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:39093 every 8 connection(s)
I20260929 18:39:41.145104 18346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:41.148242 18413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:41.149468 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18346
I20260929 18:39:41.149561 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:41.149848 792 external_mini_cluster.cc:1428] Setting key fa31da0ede60178fdb3ed9f138872e2e
I20260929 18:39:41.154290 18413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f: Bootstrap starting.
I20260929 18:39:41.156414 18413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:41.157191 18413 log.cc:824] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:41.158804 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.151358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.0.198.62:39093)
0929 18:39:41.151984 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.152003 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.152039 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:39:41.152742 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.152767 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.153162 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.153463 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:41.153881 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.154910 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.155761 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.156220 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.156361 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.157140 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:41.157145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:41.157157 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:41.157555 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:41.157608 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:41.157630 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:41.157715 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.157849 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.158114 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260929 18:39:41.159082 18413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f: No bootstrap required, opened a new log
I20260929 18:39:41.162842 18413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289f4122b4774b83ae698d4dff91c60f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39093 } }
I20260929 18:39:41.163079 18413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:41.163136 18413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289f4122b4774b83ae698d4dff91c60f, State: Initialized, Role: FOLLOWER
I20260929 18:39:41.163532 18413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289f4122b4774b83ae698d4dff91c60f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39093 } }
I20260929 18:39:41.163668 18413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:41.163728 18413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:41.163827 18413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:41.164693 18413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289f4122b4774b83ae698d4dff91c60f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39093 } }
I20260929 18:39:41.165004 18413 leader_election.cc:302] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 289f4122b4774b83ae698d4dff91c60f; no voters:
I20260929 18:39:41.165294 18413 leader_election.cc:288] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:41.165385 18418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:41.165584 18418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [term 1 LEADER]: Becoming Leader. State: Replica: 289f4122b4774b83ae698d4dff91c60f, State: Running, Role: LEADER
I20260929 18:39:41.165874 18418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289f4122b4774b83ae698d4dff91c60f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39093 } }
I20260929 18:39:41.166462 18413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:41.167517 18419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "289f4122b4774b83ae698d4dff91c60f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289f4122b4774b83ae698d4dff91c60f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39093 } } }
I20260929 18:39:41.167637 18419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:41.167563 18420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 289f4122b4774b83ae698d4dff91c60f. Latest consensus state: current_term: 1 leader_uuid: "289f4122b4774b83ae698d4dff91c60f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289f4122b4774b83ae698d4dff91c60f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 39093 } } }
I20260929 18:39:41.167840 18420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:41.168673 18427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:41.171550 18427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:41.176496 18427 catalog_manager.cc:1347] Generated new cluster ID: 9e57ff2182094704a57a50ce1c748f78
I20260929 18:39:41.176565 18427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:41.203266 18427 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:41.204552 18427 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:41.221253 18427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f: Generated new TSK 0
I20260929 18:39:41.222201 18427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:41.226428 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39093
--builtin_ntp_servers=127.0.198.20:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:39:41.338693 18437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:41.338985 18437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:41.339038 18437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:41.343130 18437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:41.343186 18437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:41.343261 18437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:41.348104 18437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:41.349184 18437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:41.350564 18437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:41.360491 18445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:41.360529 18437 server_base.cc:1034] running on GCE node
W20260929 18:39:41.360476 18442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:41.360474 18443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:41.361135 18437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:41.361814 18437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:41.362996 18437 hybrid_clock.cc:648] HybridClock initialized: now 1790707181363011 us; error 96 us; skew 500 ppm
I20260929 18:39:41.364962 18437 webserver.cc:460] Webserver started at http://127.0.198.1:38811/ using document root <none> and password file <none>
I20260929 18:39:41.365547 18437 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:41.365622 18437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:41.365808 18437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:41.368037 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "0f3fcef5eee84c11b5b8642a1edbc077"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "5dba8a97ec8cb39a0774319c944c2152"
server_key_iv: "7ad94031d49b26778635397e5eba17a2"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.368597 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "0f3fcef5eee84c11b5b8642a1edbc077"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "5dba8a97ec8cb39a0774319c944c2152"
server_key_iv: "7ad94031d49b26778635397e5eba17a2"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.372478 18437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:39:41.375036 18451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.376444 18437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:41.376566 18437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "0f3fcef5eee84c11b5b8642a1edbc077"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "5dba8a97ec8cb39a0774319c944c2152"
server_key_iv: "7ad94031d49b26778635397e5eba17a2"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.376674 18437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:41.396301 18437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:41.397006 18437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:41.397189 18437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:41.397770 18437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:41.398736 18437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:41.398792 18437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.398846 18437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:41.398880 18437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.407500 18437 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:37323
I20260929 18:39:41.407575 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:37323 every 8 connection(s)
I20260929 18:39:41.408424 18437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:41.412936 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18437
I20260929 18:39:41.413035 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:41.413324 792 external_mini_cluster.cc:1428] Setting key 7790a0bdc6a699b02d5e1bb6be660b78
I20260929 18:39:41.415444 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39093
--builtin_ntp_servers=127.0.198.20:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:39:41.418684 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.410499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:37575 (local address 127.0.198.62:39093)
0929 18:39:41.410673 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.410678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.411746 (+ 1068us) server_negotiation.cc:407] Connection header received
0929 18:39:41.413197 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.413201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.413262 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.413345 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:41.414523 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.415265 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.416592 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.416826 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.416908 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.418176 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:41.418179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:41.418181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:41.418193 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:41.418212 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:41.418220 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:41.418286 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.418486 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.418556 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:39:41.419839 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.410906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39093 (local address 127.0.198.1:37575)
0929 18:39:41.411598 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:41.411641 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:41.412962 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:41.413546 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:41.413553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:41.413644 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:41.414367 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.414386 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.415411 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.415413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.416464 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.416477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.416971 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.416974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.417181 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.417824 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:41.417826 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:41.417881 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:41.417964 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:41.418302 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:41.418311 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:41.418316 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:39:41.418448 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 18:39:41.418764 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260929 18:39:41.421216 18565 heartbeater.cc:347] Connected to a master server at 127.0.198.62:39093
I20260929 18:39:41.421456 18565 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:41.422022 18565 heartbeater.cc:511] Master 127.0.198.62:39093 requested a full tablet report, sending...
I20260929 18:39:41.423718 18375 ts_manager.cc:194] Registered new tserver with Master: 0f3fcef5eee84c11b5b8642a1edbc077 (127.0.198.1:37323)
I20260929 18:39:41.425027 18375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.1:37575
W20260929 18:39:41.526830 18568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:41.527225 18568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:41.527290 18568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:41.531533 18568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:41.531594 18568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:41.531651 18568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:41.536552 18568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:41.537746 18568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:41.539179 18568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:41.548866 18573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:41.548945 18568 server_base.cc:1034] running on GCE node
W20260929 18:39:41.548880 18574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:41.548887 18576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:41.549582 18568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:41.550274 18568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:41.551456 18568 hybrid_clock.cc:648] HybridClock initialized: now 1790707181551429 us; error 41 us; skew 500 ppm
I20260929 18:39:41.553273 18568 webserver.cc:460] Webserver started at http://127.0.198.2:35331/ using document root <none> and password file <none>
I20260929 18:39:41.553835 18568 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:41.553902 18568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:41.554121 18568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:41.556201 18568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "8bcf481097e741e4a331ae3a28ded7f0"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "a1839676fc92f92b0c60383bde2d7737"
server_key_iv: "09bbaeca331e5b98f882274cabe1bdf8"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.556764 18568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "8bcf481097e741e4a331ae3a28ded7f0"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "a1839676fc92f92b0c60383bde2d7737"
server_key_iv: "09bbaeca331e5b98f882274cabe1bdf8"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.560791 18568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:39:41.563598 18582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.564998 18568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:39:41.565129 18568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "8bcf481097e741e4a331ae3a28ded7f0"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "a1839676fc92f92b0c60383bde2d7737"
server_key_iv: "09bbaeca331e5b98f882274cabe1bdf8"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.565316 18568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:41.604360 18568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:41.605064 18568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:41.605245 18568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:41.605821 18568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:41.606787 18568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:41.606842 18568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.606896 18568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:41.606922 18568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.615104 18568 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:36419
I20260929 18:39:41.615151 18695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:36419 every 8 connection(s)
I20260929 18:39:41.615967 18568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:41.622400 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18568
I20260929 18:39:41.622516 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:39:41.622813 792 external_mini_cluster.cc:1428] Setting key 8ba9bc5cd6b8d301264a1211f4075d1d
I20260929 18:39:41.624949 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39093
--builtin_ntp_servers=127.0.198.20:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:39:41.626173 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.618164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:56703 (local address 127.0.198.62:39093)
0929 18:39:41.618300 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.618304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.619473 (+ 1169us) server_negotiation.cc:407] Connection header received
0929 18:39:41.620859 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.620863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.620921 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.620994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:41.621947 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.622705 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.624181 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.624431 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.624542 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.625617 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:41.625621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:41.625623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:41.625638 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:41.625660 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:41.625670 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:41.625745 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.625932 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.626024 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:39:41.627069 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.618572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39093 (local address 127.0.198.2:56703)
0929 18:39:41.619309 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:41.619353 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:41.620643 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:41.621174 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:41.621181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:41.621275 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:41.621813 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.621826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.622856 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.622861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.624052 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.624063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.624593 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.624596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.624753 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.625343 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:41.625345 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:41.625387 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:41.625455 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:41.625784 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:41.625793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:41.625800 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:39:41.625926 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:39:41.626240 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
I20260929 18:39:41.628351 18696 heartbeater.cc:347] Connected to a master server at 127.0.198.62:39093
I20260929 18:39:41.628592 18696 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:41.629191 18696 heartbeater.cc:511] Master 127.0.198.62:39093 requested a full tablet report, sending...
I20260929 18:39:41.630225 18375 ts_manager.cc:194] Registered new tserver with Master: 8bcf481097e741e4a331ae3a28ded7f0 (127.0.198.2:36419)
I20260929 18:39:41.630779 18375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.2:56703
W20260929 18:39:41.734618 18699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:41.734880 18699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:41.734933 18699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:41.738950 18699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:41.739008 18699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:41.739089 18699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:41.744038 18699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:41.745210 18699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:41.746610 18699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:41.755795 18704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:41.755714 18707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:41.755710 18705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:41.755766 18699 server_base.cc:1034] running on GCE node
I20260929 18:39:41.756330 18699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:41.756917 18699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:41.758076 18699 hybrid_clock.cc:648] HybridClock initialized: now 1790707181758043 us; error 41 us; skew 500 ppm
I20260929 18:39:41.759920 18699 webserver.cc:460] Webserver started at http://127.0.198.3:36803/ using document root <none> and password file <none>
I20260929 18:39:41.760460 18699 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:41.760550 18699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:41.760751 18699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:41.762813 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "ee53b019c0de4f51a1ceb30f76187aaf"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "26f41c348666dfd184c05bbb5968b742"
server_key_iv: "b60b500c4bf6c8e15a6f299d38d2bd83"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.763345 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "ee53b019c0de4f51a1ceb30f76187aaf"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "26f41c348666dfd184c05bbb5968b742"
server_key_iv: "b60b500c4bf6c8e15a6f299d38d2bd83"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.766988 18699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:41.769670 18713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.770916 18699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:41.771028 18699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "ee53b019c0de4f51a1ceb30f76187aaf"
format_stamp: "Formatted at 2026-09-29 18:39:41 on dist-test-slave-q5qx"
server_key: "26f41c348666dfd184c05bbb5968b742"
server_key_iv: "b60b500c4bf6c8e15a6f299d38d2bd83"
server_key_version: "encryptionkey@0"
I20260929 18:39:41.771124 18699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:41.789906 18699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:41.790637 18699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:41.790812 18699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:41.791388 18699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:41.792320 18699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:41.792364 18699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.792415 18699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:41.792443 18699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:41.800768 18699 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:39471
I20260929 18:39:41.800838 18826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:39471 every 8 connection(s)
I20260929 18:39:41.801702 18699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:41.810737 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.803550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:41221 (local address 127.0.198.62:39093)
0929 18:39:41.803695 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.803698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.804740 (+ 1042us) server_negotiation.cc:407] Connection header received
0929 18:39:41.806128 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.806132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.806199 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.806272 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:41.807150 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.807688 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.808837 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.809084 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.809174 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.810275 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:41.810279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:41.810282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:41.810296 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:41.810315 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:41.810322 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:41.810380 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.810513 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.810576 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:39:41.811553 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.803938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:39093 (local address 127.0.198.3:41221)
0929 18:39:41.804590 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:41.804630 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:41.805902 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:41.806443 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:41.806450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:41.806535 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:41.807015 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.807028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.807806 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.807809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.808716 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.808726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.809241 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.809245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.809403 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.810015 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:41.810016 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:41.810058 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:41.810124 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:41.810390 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:41.810398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:41.810404 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:39:41.810509 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 18:39:41.810783 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260929 18:39:41.811939 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18699
I20260929 18:39:41.812023 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:41.812282 792 external_mini_cluster.cc:1428] Setting key 0cde361eac4cf5fbaeea719173429d68
I20260929 18:39:41.812704 18827 heartbeater.cc:347] Connected to a master server at 127.0.198.62:39093
I20260929 18:39:41.812942 18827 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:41.813504 18827 heartbeater.cc:511] Master 127.0.198.62:39093 requested a full tablet report, sending...
I20260929 18:39:41.814509 18375 ts_manager.cc:194] Registered new tserver with Master: ee53b019c0de4f51a1ceb30f76187aaf (127.0.198.3:39471)
I20260929 18:39:41.815047 18375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.3:41221
I20260929 18:39:41.826289 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:41.831941 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.828287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.0.198.62:39093)
0929 18:39:41.828417 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.828422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.828542 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:39:41.828685 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.828687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.828728 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.828786 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:41.829107 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.829641 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.830491 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.830704 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.830783 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.831577 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:41.831580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:41.831582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:41.831593 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:41.831612 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:41.831620 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:41.831679 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.831744 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.831793 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:39:41.834884 18375 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:41.837913 18375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:41.851756 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.846178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:36419 (local address 127.0.0.1:49468)
0929 18:39:41.846518 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:41.846569 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:41.846740 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:41.847637 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:41.847643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:41.847665 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:41.847801 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.847808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.849028 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.849031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.850296 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.850306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.851211 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.851215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.851316 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.851358 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:39:41.851469 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:39:41.851604 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260929 18:39:41.851774 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.846327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.198.2:36419)
0929 18:39:41.847135 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.847148 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.847173 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:39:41.847247 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.847253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.847468 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.847628 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:41.847906 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.848914 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.850427 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.851087 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.851204 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.851408 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.851499 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.851604 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":169,"threads_started":1}
I20260929 18:39:41.852370 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.846488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37323 (local address 127.0.0.1:42060)
0929 18:39:41.847502 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:41.847516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:41.847608 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:41.848172 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:41.848175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:41.848194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:41.848286 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.848292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.849424 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.849428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.850629 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.850639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.851977 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.851981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.852070 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.852094 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:39:41.852170 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:39:41.852244 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":97,"threads_started":1}
I20260929 18:39:41.852479 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.846884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.0.198.1:37323)
0929 18:39:41.847371 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.847383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.847717 (+ 334us) server_negotiation.cc:407] Connection header received
0929 18:39:41.847800 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.847805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.847982 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.848198 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:41.848407 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.849312 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.850758 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.851500 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.851592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.851773 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.852226 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.852328 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":141,"threads_started":1}
I20260929 18:39:41.852849 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.847288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.0.198.3:39471)
0929 18:39:41.847895 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:41.847906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:41.847925 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:39:41.847987 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:41.847991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:41.848151 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:41.848289 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:41.848657 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.849518 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.850883 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.851565 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.851656 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.851826 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:41.852597 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 18:39:41.852702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":82,"threads_started":1}
I20260929 18:39:41.853143 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:41.846888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39471 (local address 127.0.0.1:52272)
0929 18:39:41.847605 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:41.847616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:41.847680 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:41.848426 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:41.848429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:41.848449 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:41.848537 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.848543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.849671 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.849674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.850784 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:41.850791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:41.852126 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:41.852129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:41.852216 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:41.852242 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:39:41.852308 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:39:41.852372 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":82,"threads_started":1}
I20260929 18:39:41.855120 18499 tablet_service.cc:1467] Processing CreateTablet for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab (DEFAULT_TABLE table=test-table [id=3cfce27f1d134835b25be8c95dabbec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:41.855187 18630 tablet_service.cc:1467] Processing CreateTablet for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab (DEFAULT_TABLE table=test-table [id=3cfce27f1d134835b25be8c95dabbec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:41.856094 18499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:41.855759 18761 tablet_service.cc:1467] Processing CreateTablet for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab (DEFAULT_TABLE table=test-table [id=3cfce27f1d134835b25be8c95dabbec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:41.856216 18630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:41.856632 18761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:41.862371 18844 tablet_bootstrap.cc:492] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077: Bootstrap starting.
I20260929 18:39:41.864271 18846 tablet_bootstrap.cc:492] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Bootstrap starting.
I20260929 18:39:41.864560 18844 tablet_bootstrap.cc:654] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:41.865233 18844 log.cc:824] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:41.865847 18845 tablet_bootstrap.cc:492] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Bootstrap starting.
I20260929 18:39:41.866171 18846 tablet_bootstrap.cc:654] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:41.866997 18846 log.cc:824] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:41.867399 18844 tablet_bootstrap.cc:492] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077: No bootstrap required, opened a new log
I20260929 18:39:41.867586 18844 ts_tablet_manager.cc:1397] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:39:41.867765 18845 tablet_bootstrap.cc:654] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:41.868490 18845 log.cc:824] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:41.869731 18846 tablet_bootstrap.cc:492] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: No bootstrap required, opened a new log
I20260929 18:39:41.869894 18846 ts_tablet_manager.cc:1397] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:39:41.870510 18845 tablet_bootstrap.cc:492] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: No bootstrap required, opened a new log
I20260929 18:39:41.870692 18845 ts_tablet_manager.cc:1397] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:41.871887 18844 raft_consensus.cc:348] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:41.872160 18844 raft_consensus.cc:374] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:41.872212 18844 raft_consensus.cc:729] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3fcef5eee84c11b5b8642a1edbc077, State: Initialized, Role: FOLLOWER
I20260929 18:39:41.872658 18844 consensus_queue.cc:260] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:41.873109 18846 raft_consensus.cc:348] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:41.873356 18846 raft_consensus.cc:374] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:41.873412 18846 raft_consensus.cc:729] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee53b019c0de4f51a1ceb30f76187aaf, State: Initialized, Role: FOLLOWER
I20260929 18:39:41.873560 18565 heartbeater.cc:503] Master 127.0.198.62:39093 was elected leader, sending a full tablet report...
I20260929 18:39:41.873670 18844 ts_tablet_manager.cc:1428] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:39:41.873840 18846 consensus_queue.cc:260] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:41.874963 18846 ts_tablet_manager.cc:1428] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:39:41.874846 18845 raft_consensus.cc:348] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:41.875308 18827 heartbeater.cc:503] Master 127.0.198.62:39093 was elected leader, sending a full tablet report...
I20260929 18:39:41.875381 18845 raft_consensus.cc:374] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:41.875433 18845 raft_consensus.cc:729] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcf481097e741e4a331ae3a28ded7f0, State: Initialized, Role: FOLLOWER
I20260929 18:39:41.875870 18845 consensus_queue.cc:260] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:41.876528 18696 heartbeater.cc:503] Master 127.0.198.62:39093 was elected leader, sending a full tablet report...
I20260929 18:39:41.877440 18845 ts_tablet_manager.cc:1428] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260929 18:39:41.909683 18566 tablet.cc:2369] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:39:42.052724 18828 tablet.cc:2369] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:42.080641 18850 raft_consensus.cc:482] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:42.080919 18850 raft_consensus.cc:504] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:42.082281 18850 leader_election.cc:288] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bcf481097e741e4a331ae3a28ded7f0 (127.0.198.2:36419), ee53b019c0de4f51a1ceb30f76187aaf (127.0.198.3:39471)
I20260929 18:39:42.087441 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:42.082982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:36823 (local address 127.0.198.3:39471)
0929 18:39:42.083131 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:42.083135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:42.083272 (+ 137us) server_negotiation.cc:407] Connection header received
0929 18:39:42.083371 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:42.083374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:42.083415 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:42.083479 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:42.083704 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.084398 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.086035 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.086695 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.086812 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:42.086880 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:42.087234 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 18:39:42.087313 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:39:42.087441 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:42.082637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:33503 (local address 127.0.198.2:36419)
0929 18:39:42.082760 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:42.082764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:42.082933 (+ 169us) server_negotiation.cc:407] Connection header received
0929 18:39:42.082992 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:42.082995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:42.083183 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:42.083270 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:42.083516 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.084290 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.086132 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.086701 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.086812 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:42.086881 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:42.087234 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 18:39:42.087312 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:39:42.087468 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:42.082861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:39471 (local address 127.0.198.1:36823)
0929 18:39:42.083187 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:42.083199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:42.083289 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:42.083501 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:42.083505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:42.083521 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:42.083616 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:42.083620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.084497 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.084501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:42.085868 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:42.085881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.086886 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.086889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:42.087056 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:42.087118 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 18:39:42.087191 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:39:42.087283 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":122,"thread_start_us":66,"threads_started":1}
I20260929 18:39:42.087471 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:42.082510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:36419 (local address 127.0.198.1:33503)
0929 18:39:42.082748 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:42.082763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:42.082864 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:42.083287 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:42.083290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:42.083303 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:42.083416 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:42.083421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.084414 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.084418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:42.085868 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:42.085881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.086881 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.086889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:42.087072 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:42.087106 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:39:42.087200 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:39:42.087283 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":117,"thread_start_us":76,"threads_started":1}
I20260929 18:39:42.088233 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe37efaad54e5dbcb0a4362a2ab4ab" candidate_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" is_pre_election: true
I20260929 18:39:42.088411 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe37efaad54e5dbcb0a4362a2ab4ab" candidate_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bcf481097e741e4a331ae3a28ded7f0" is_pre_election: true
I20260929 18:39:42.088580 18781 raft_consensus.cc:2454] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3fcef5eee84c11b5b8642a1edbc077 in term 0.
I20260929 18:39:42.088673 18650 raft_consensus.cc:2454] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3fcef5eee84c11b5b8642a1edbc077 in term 0.
I20260929 18:39:42.089078 18452 leader_election.cc:302] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3fcef5eee84c11b5b8642a1edbc077, ee53b019c0de4f51a1ceb30f76187aaf; no voters:
I20260929 18:39:42.089329 18850 raft_consensus.cc:2781] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:42.089414 18850 raft_consensus.cc:482] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:42.089444 18850 raft_consensus.cc:3037] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:42.090485 18850 raft_consensus.cc:504] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:42.090843 18850 leader_election.cc:288] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [CANDIDATE]: Term 1 election: Requested vote from peers 8bcf481097e741e4a331ae3a28ded7f0 (127.0.198.2:36419), ee53b019c0de4f51a1ceb30f76187aaf (127.0.198.3:39471)
I20260929 18:39:42.091226 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe37efaad54e5dbcb0a4362a2ab4ab" candidate_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bcf481097e741e4a331ae3a28ded7f0"
I20260929 18:39:42.091280 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe37efaad54e5dbcb0a4362a2ab4ab" candidate_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee53b019c0de4f51a1ceb30f76187aaf"
I20260929 18:39:42.091344 18650 raft_consensus.cc:3037] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:42.091392 18781 raft_consensus.cc:3037] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:42.092386 18781 raft_consensus.cc:2454] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3fcef5eee84c11b5b8642a1edbc077 in term 1.
I20260929 18:39:42.092417 18650 raft_consensus.cc:2454] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3fcef5eee84c11b5b8642a1edbc077 in term 1.
I20260929 18:39:42.092667 18452 leader_election.cc:302] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3fcef5eee84c11b5b8642a1edbc077, ee53b019c0de4f51a1ceb30f76187aaf; no voters:
I20260929 18:39:42.092833 18850 raft_consensus.cc:2781] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:42.093040 18850 raft_consensus.cc:686] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3fcef5eee84c11b5b8642a1edbc077, State: Running, Role: LEADER
I20260929 18:39:42.093374 18850 consensus_queue.cc:237] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } }
I20260929 18:39:42.096715 18375 catalog_manager.cc:5495] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3fcef5eee84c11b5b8642a1edbc077 (127.0.198.1). New cstate: current_term: 1 leader_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f3fcef5eee84c11b5b8642a1edbc077" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37323 } health_report { overall_health: HEALTHY } } }
W20260929 18:39:42.117334 18697 tablet.cc:2369] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:42.147948 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:42.143817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.0.198.1:37323)
0929 18:39:42.143990 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:42.143996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:42.144014 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:39:42.144106 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:42.144109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:42.144246 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:42.144393 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:42.144677 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.145400 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.146364 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:42.146691 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:42.146797 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:42.147066 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:42.147561 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:42.147678 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:42.147760 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 18:39:42.147816 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:39:42.154341 18781 raft_consensus.cc:1261] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 1 FOLLOWER]: Refusing update from remote peer 0f3fcef5eee84c11b5b8642a1edbc077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:42.154341 18650 raft_consensus.cc:1261] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 1 FOLLOWER]: Refusing update from remote peer 0f3fcef5eee84c11b5b8642a1edbc077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:42.155052 18850 consensus_queue.cc:1035] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bcf481097e741e4a331ae3a28ded7f0" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:42.155256 18855 consensus_queue.cc:1035] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 0f3fcef5eee84c11b5b8642a1edbc077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee53b019c0de4f51a1ceb30f76187aaf" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:42.162225 18859 mvcc.cc:205] Tried to move back new op lower bound from 7334736618095988736 to 7334736617857007616. Current Snapshot: MvccSnapshot[applied={T|T < 7334736618095988736}]
I20260929 18:39:42.170368 18377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:42.170526 18377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:42.172859 18377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 289f4122b4774b83ae698d4dff91c60f: Sending DeleteTablet for 3 replicas of tablet 66fe37efaad54e5dbcb0a4362a2ab4ab
I20260929 18:39:42.173687 18761 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:42 UTC) from {username='slave'} at 127.0.0.1:52272
I20260929 18:39:42.173729 18499 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:42 UTC) from {username='slave'} at 127.0.0.1:42060
I20260929 18:39:42.173729 18630 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:42 UTC) from {username='slave'} at 127.0.0.1:49468
I20260929 18:39:42.174201 18868 tablet_replica.cc:330] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: stopping tablet replica
I20260929 18:39:42.174391 18870 tablet_replica.cc:330] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: stopping tablet replica
I20260929 18:39:42.174681 18868 raft_consensus.cc:2229] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:42.174690 18870 raft_consensus.cc:2229] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:42.174832 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18437
I20260929 18:39:42.176043 18868 raft_consensus.cc:2258] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:42.176632 18870 raft_consensus.cc:2258] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:42.177764 18868 ts_tablet_manager.cc:1905] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:42.178052 18870 ts_tablet_manager.cc:1905] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:42.180470 18870 ts_tablet_manager.cc:1918] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:42.180470 18868 ts_tablet_manager.cc:1918] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:42.180572 18870 log.cc:1196] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/66fe37efaad54e5dbcb0a4362a2ab4ab
I20260929 18:39:42.180572 18868 log.cc:1196] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/66fe37efaad54e5dbcb0a4362a2ab4ab
I20260929 18:39:42.180861 18870 ts_tablet_manager.cc:1939] T 66fe37efaad54e5dbcb0a4362a2ab4ab P 8bcf481097e741e4a331ae3a28ded7f0: Deleting consensus metadata
I20260929 18:39:42.180861 18868 ts_tablet_manager.cc:1939] T 66fe37efaad54e5dbcb0a4362a2ab4ab P ee53b019c0de4f51a1ceb30f76187aaf: Deleting consensus metadata
W20260929 18:39:42.180912 18362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:37323 (error 108)
I20260929 18:39:42.181659 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18568
I20260929 18:39:42.182989 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:42.182001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37323 (local address 127.0.0.1:42078)
0929 18:39:42.182854 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:42.182915 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":776}
I20260929 18:39:42.183346 18361 catalog_manager.cc:4841] TS ee53b019c0de4f51a1ceb30f76187aaf (127.0.198.3:39471): tablet 66fe37efaad54e5dbcb0a4362a2ab4ab (table test-table [id=3cfce27f1d134835b25be8c95dabbec8]) successfully deleted
W20260929 18:39:42.183617 18362 catalog_manager.cc:4584] TS 0f3fcef5eee84c11b5b8642a1edbc077 (127.0.198.1:37323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66fe37efaad54e5dbcb0a4362a2ab4ab: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37323: connect: Connection refused (error 111)
I20260929 18:39:42.187260 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18699
I20260929 18:39:42.188243 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:42.187455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:36419 (local address 127.0.0.1:49470)
0929 18:39:42.188000 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:42.188155 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.2:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453}
I20260929 18:39:42.192502 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18346
2026-09-29T18:39:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1245 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:42 dist-test-slave-q5qx krb5kdc[18874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:42 dist-test-slave-q5qx krb5kdc[18874](info): set up 2 sockets
Sep 29 18:39:42 dist-test-slave-q5qx krb5kdc[18874](info): commencing operation
krb5kdc: starting...
W20260929 18:39:43.153606 792 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:43 dist-test-slave-q5qx krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:39:43.277251 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42329
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:43.385479 18887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:43.385766 18887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:43.385819 18887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:43.391291 18887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:43.391359 18887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:43.391384 18887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:43.391407 18887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:43.391435 18887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:43.397828 18887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:43.399276 18887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:43.400414 18887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:43.408977 18895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:43.408982 18893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:43.408982 18892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:43.409394 18887 server_base.cc:1034] running on GCE node
I20260929 18:39:43.409935 18887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:43.411098 18887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:43.412271 18887 hybrid_clock.cc:648] HybridClock initialized: now 1790707183412248 us; error 40 us; skew 500 ppm
Sep 29 18:39:43 dist-test-slave-q5qx krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:43.414995 18887 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:39:43.415969 18887 webserver.cc:460] Webserver started at http://127.0.198.62:39291/ using document root <none> and password file <none>
I20260929 18:39:43.416486 18887 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:43.416543 18887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:43.416714 18887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:43.418812 18887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "9391cc2847e54b3ca94eba56c37ddb05"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "3d56a515286aff4a3ef6efda1154b6d1"
server_key_iv: "10ef3e229848329bb1fe64d9c057d432"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.419327 18887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "9391cc2847e54b3ca94eba56c37ddb05"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "3d56a515286aff4a3ef6efda1154b6d1"
server_key_iv: "10ef3e229848329bb1fe64d9c057d432"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.423235 18887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 18:39:43.425745 18902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:43.427184 18887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:43.427302 18887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "9391cc2847e54b3ca94eba56c37ddb05"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "3d56a515286aff4a3ef6efda1154b6d1"
server_key_iv: "10ef3e229848329bb1fe64d9c057d432"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.427402 18887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:43.442170 18887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:43.445061 18887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:43.445218 18887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:43.451891 18887 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:42329
I20260929 18:39:43.451952 18954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:42329 every 8 connection(s)
I20260929 18:39:43.452801 18887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:43.453482 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18887
I20260929 18:39:43.453614 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:43.453912 792 external_mini_cluster.cc:1428] Setting key 177c8f3f0240d56014dcc5f03b7e9cfb
I20260929 18:39:43.455952 18955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:43.462297 18955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05: Bootstrap starting.
Sep 29 18:39:43 dist-test-slave-q5qx krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:43.464715 18955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:43.465422 18955 log.cc:824] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:43.467729 18955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05: No bootstrap required, opened a new log
I20260929 18:39:43.468596 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:43.455235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.0.198.62:42329)
0929 18:39:43.455616 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:43.455631 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:43.455664 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:39:43.456386 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:43.456405 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:43.456736 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:43.457011 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:43.457884 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.458958 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.460103 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.460584 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.460708 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:43.463559 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:43.463564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:43.463577 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:43.463605 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:43.465336 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:43.465837 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:43.465844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:43.465850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:43.466039 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:43.466396 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:43.466399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:43.466400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:43.466816 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:43.466979 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:43.467320 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 18:39:43.467611 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":114,"threads_started":1}
I20260929 18:39:43.470971 18955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9391cc2847e54b3ca94eba56c37ddb05" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:39:43.471307 18955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:43.471374 18955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9391cc2847e54b3ca94eba56c37ddb05, State: Initialized, Role: FOLLOWER
I20260929 18:39:43.471805 18955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9391cc2847e54b3ca94eba56c37ddb05" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:39:43.471928 18955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:43.472231 18955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:43.472359 18955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:43.473336 18955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9391cc2847e54b3ca94eba56c37ddb05" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:39:43.473670 18955 leader_election.cc:302] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9391cc2847e54b3ca94eba56c37ddb05; no voters:
I20260929 18:39:43.474020 18955 leader_election.cc:288] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:43.474195 18960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:43.474398 18960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [term 1 LEADER]: Becoming Leader. State: Replica: 9391cc2847e54b3ca94eba56c37ddb05, State: Running, Role: LEADER
I20260929 18:39:43.474723 18960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9391cc2847e54b3ca94eba56c37ddb05" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } }
I20260929 18:39:43.475203 18955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:43.476251 18962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9391cc2847e54b3ca94eba56c37ddb05. Latest consensus state: current_term: 1 leader_uuid: "9391cc2847e54b3ca94eba56c37ddb05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9391cc2847e54b3ca94eba56c37ddb05" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } } }
I20260929 18:39:43.476383 18962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:43.476765 18969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:43.476755 18961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9391cc2847e54b3ca94eba56c37ddb05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9391cc2847e54b3ca94eba56c37ddb05" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42329 } } }
I20260929 18:39:43.476907 18961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:43.479998 18969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:43.486490 18969 catalog_manager.cc:1347] Generated new cluster ID: 2f148e3cd1e54c4b9c33e324c978201c
I20260929 18:39:43.486570 18969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:43.500569 18969 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:43.501569 18969 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:43.506232 18969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05: Generated new TSK 0
I20260929 18:39:43.506886 18969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:39:43.537627 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:43.651081 18981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:43.651352 18981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:43.651394 18981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:43.655558 18981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:43.655620 18981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:43.655684 18981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:43.660988 18981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:43.662058 18981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:43.663090 18981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:43.672709 18986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:43.672776 18989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:43.672756 18981 server_base.cc:1034] running on GCE node
W20260929 18:39:43.672704 18987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:43.673346 18981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:43.673993 18981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:43.675195 18981 hybrid_clock.cc:648] HybridClock initialized: now 1790707183675177 us; error 32 us; skew 500 ppm
Sep 29 18:39:43 dist-test-slave-q5qx krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:43.678102 18981 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:39:43.679018 18981 webserver.cc:460] Webserver started at http://127.0.198.1:43407/ using document root <none> and password file <none>
I20260929 18:39:43.679553 18981 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:43.679615 18981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:43.679795 18981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:43.681835 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "3f49d95bc4c1411badff79dd28b74f1e"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "7c4b9efbb16182e26f964f7a23a18a86"
server_key_iv: "0bb394493ac9757ec858f55a767bc8f5"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.682421 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "3f49d95bc4c1411badff79dd28b74f1e"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "7c4b9efbb16182e26f964f7a23a18a86"
server_key_iv: "0bb394493ac9757ec858f55a767bc8f5"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.686158 18981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:39:43.688578 18996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:43.689854 18981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:43.689961 18981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "3f49d95bc4c1411badff79dd28b74f1e"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "7c4b9efbb16182e26f964f7a23a18a86"
server_key_iv: "0bb394493ac9757ec858f55a767bc8f5"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.690091 18981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:43.706404 18981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:43.709388 18981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:43.709568 18981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:43.710177 18981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:43.710978 18981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:43.711035 18981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:43.711088 18981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:43.711112 18981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:43.719672 18981 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:37915
I20260929 18:39:43.719709 19109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:37915 every 8 connection(s)
I20260929 18:39:43.720549 18981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:43.724010 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 18981
I20260929 18:39:43.724125 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:43.724436 792 external_mini_cluster.cc:1428] Setting key 5661b4d19b4ba8c845bc6550098ba0ac
Sep 29 18:39:43 dist-test-slave-q5qx krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:39:43.736379 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:43.722485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:33609 (local address 127.0.198.62:42329)
0929 18:39:43.722652 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:43.722657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:43.723569 (+ 912us) server_negotiation.cc:407] Connection header received
0929 18:39:43.724974 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:43.724978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:43.725032 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:43.725117 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:43.726683 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.727422 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.729012 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.729436 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.729546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:43.732285 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:43.732289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:43.732292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:43.732323 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:43.733895 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:43.734556 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:43.734559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:43.734561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:43.734671 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:43.735131 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:43.735134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:43.735138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:43.735383 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:43.735470 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:43.736084 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 18:39:43.736251 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:39:43.737416 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:43.722908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:42329 (local address 127.0.198.1:33609)
0929 18:39:43.723416 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:43.723461 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:43.724722 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:43.725349 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:43.725359 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:43.725855 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:43.726533 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:43.726547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.727573 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.727577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:43.728880 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:43.728891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.729633 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.729636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:43.729798 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:43.730517 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:43.730545 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:43.732096 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:43.734079 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:43.734085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:43.734101 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:43.734425 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:43.734794 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:43.734797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:43.734799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:43.734980 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:43.735510 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:43.735516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:43.735837 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 18:39:43.736029 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 18:39:43.736454 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260929 18:39:43.738929 19110 heartbeater.cc:347] Connected to a master server at 127.0.198.62:42329
I20260929 18:39:43.739179 19110 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:43.739771 19110 heartbeater.cc:511] Master 127.0.198.62:42329 requested a full tablet report, sending...
I20260929 18:39:43.741706 18919 ts_manager.cc:194] Registered new tserver with Master: 3f49d95bc4c1411badff79dd28b74f1e (127.0.198.1:37915)
I20260929 18:39:43.743355 18919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:33609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:39:43.753131 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:43.863044 19115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:43.863334 19115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:43.863379 19115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:43.867475 19115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:43.867540 19115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:43.867599 19115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:43.873051 19115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:43.874089 19115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:43.875102 19115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:43.884871 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:43.884859 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:43.884864 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:43.885268 19115 server_base.cc:1034] running on GCE node
I20260929 18:39:43.885645 19115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:43.886312 19115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:43.887485 19115 hybrid_clock.cc:648] HybridClock initialized: now 1790707183887454 us; error 48 us; skew 500 ppm
Sep 29 18:39:43 dist-test-slave-q5qx krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:43.916407 19115 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:39:43.917465 19115 webserver.cc:460] Webserver started at http://127.0.198.2:40867/ using document root <none> and password file <none>
I20260929 18:39:43.918103 19115 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:43.918182 19115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:43.918390 19115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:43.920634 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "4dcb3b32200b4d619d4cfaf154a7e543"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "067d79c897e04d662930eb808923190e"
server_key_iv: "b35dfa469472600bdecf012dbf222016"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.921223 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "4dcb3b32200b4d619d4cfaf154a7e543"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "067d79c897e04d662930eb808923190e"
server_key_iv: "b35dfa469472600bdecf012dbf222016"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.925318 19115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:39:43.928067 19130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:43.929509 19115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:43.929625 19115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "4dcb3b32200b4d619d4cfaf154a7e543"
format_stamp: "Formatted at 2026-09-29 18:39:43 on dist-test-slave-q5qx"
server_key: "067d79c897e04d662930eb808923190e"
server_key_iv: "b35dfa469472600bdecf012dbf222016"
server_key_version: "encryptionkey@0"
I20260929 18:39:43.929744 19115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:43.941356 19115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:43.944499 19115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:43.944671 19115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:43.945254 19115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:43.946110 19115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:43.946154 19115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:43.946208 19115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:43.946237 19115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:43.954566 19115 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:37597
I20260929 18:39:43.954592 19243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:37597 every 8 connection(s)
I20260929 18:39:43.955441 19115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:43.959790 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 19115
I20260929 18:39:43.959920 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:39:43.960223 792 external_mini_cluster.cc:1428] Setting key 2c5753e2bdca674c031ac1aaa3093324
Sep 29 18:39:43 dist-test-slave-q5qx krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
I20260929 18:39:43.969396 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:43.957097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:47307 (local address 127.0.198.62:42329)
0929 18:39:43.957253 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:43.957258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:43.958074 (+ 816us) server_negotiation.cc:407] Connection header received
0929 18:39:43.959302 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:43.959305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:43.959352 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:43.959421 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:43.960874 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.961389 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.962596 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.962815 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.962903 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:43.965638 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:43.965641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:43.965643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:43.965668 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:43.967051 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:43.967667 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:43.967671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:43.967673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:43.967777 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:43.968192 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:43.968195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:43.968196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:43.968388 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:43.968470 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:43.969168 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 18:39:43.969274 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:39:43.970259 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:43.957484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:42329 (local address 127.0.198.2:47307)
0929 18:39:43.957915 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:43.957957 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:43.959107 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:43.959598 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:43.959607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:43.960121 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:43.960736 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:43.960758 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.961505 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.961508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:43.962483 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:43.962493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:43.962951 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:43.962955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:43.963118 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:43.963753 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:43.963770 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:43.965440 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:43.967168 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:43.967174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:43.967200 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:43.967546 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:43.967884 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:43.967887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:43.967889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:43.968069 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:43.968499 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:43.968507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:43.968835 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 18:39:43.969049 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 18:39:43.969395 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
I20260929 18:39:43.971431 19244 heartbeater.cc:347] Connected to a master server at 127.0.198.62:42329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:39:43.971727 19244 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:43.972275 19244 heartbeater.cc:511] Master 127.0.198.62:42329 requested a full tablet report, sending...
I20260929 18:39:43.973425 18919 ts_manager.cc:194] Registered new tserver with Master: 4dcb3b32200b4d619d4cfaf154a7e543 (127.0.198.2:37597)
I20260929 18:39:43.974035 18919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:47307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:39:43.989015 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:44.099229 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:44.099506 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:44.099550 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:44.103641 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:44.103704 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:44.103765 19249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:44.109133 19249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:44.110235 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:44.111246 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:44.120853 19254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:44.120887 19257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:44.120887 19255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:44.121007 19249 server_base.cc:1034] running on GCE node
I20260929 18:39:44.121527 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:44.122284 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:44.123459 19249 hybrid_clock.cc:648] HybridClock initialized: now 1790707184123422 us; error 50 us; skew 500 ppm
Sep 29 18:39:44 dist-test-slave-q5qx krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:44.126349 19249 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:39:44.127282 19249 webserver.cc:460] Webserver started at http://127.0.198.3:34507/ using document root <none> and password file <none>
I20260929 18:39:44.127861 19249 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:44.127933 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:44.128125 19249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:44.130296 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "1acf2831daeb433a9a6432588e8fbc5d"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "8ccc037ba5c96af6bc807051abd12b1c"
server_key_iv: "8b03aa7725f681f5675cbd51f4119e90"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.130831 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "1acf2831daeb433a9a6432588e8fbc5d"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "8ccc037ba5c96af6bc807051abd12b1c"
server_key_iv: "8b03aa7725f681f5675cbd51f4119e90"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.134660 19249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:39:44.137195 19264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.138557 19249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:44.138680 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "1acf2831daeb433a9a6432588e8fbc5d"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "8ccc037ba5c96af6bc807051abd12b1c"
server_key_iv: "8b03aa7725f681f5675cbd51f4119e90"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.138793 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:44.153296 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:44.156064 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:44.156234 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:44.156811 19249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:44.157610 19249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:44.157656 19249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.157709 19249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:44.157737 19249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.166436 19249 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:42631
I20260929 18:39:44.166458 19377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:42631 every 8 connection(s)
I20260929 18:39:44.167317 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:44.175714 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 19249
I20260929 18:39:44.175827 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:44.176151 792 external_mini_cluster.cc:1428] Setting key a6e629518fe340dc96aa5a7b81fb0136
Sep 29 18:39:44 dist-test-slave-q5qx krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:44.180286 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.169044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:42099 (local address 127.0.198.62:42329)
0929 18:39:44.169171 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.169175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.169991 (+ 816us) server_negotiation.cc:407] Connection header received
0929 18:39:44.171340 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.171344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.171400 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.171472 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:44.172633 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.173154 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.174289 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.174509 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.174599 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.177210 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:44.177214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:44.177217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:44.177241 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:44.178617 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:44.179069 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:44.179072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:44.179073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:44.179141 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:44.179412 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:44.179414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:44.179415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:44.179575 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:44.179644 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.180070 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.180163 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:39:44.181071 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.169410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:42329 (local address 127.0.198.3:42099)
0929 18:39:44.169844 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.169886 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:44.171093 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:44.171652 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:44.171664 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:44.172019 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:44.172484 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.172498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.173271 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.173274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.174168 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.174177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.174647 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.174651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.174815 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.175339 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:44.175354 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:44.177064 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:44.178723 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:44.178728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:44.178739 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:44.178995 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:44.179213 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:44.179216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:44.179217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:44.179345 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:44.179645 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:44.179649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:44.179875 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 18:39:44.180023 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 18:39:44.180300 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260929 18:39:44.182133 19378 heartbeater.cc:347] Connected to a master server at 127.0.198.62:42329
I20260929 18:39:44.182379 19378 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:44.182830 19378 heartbeater.cc:511] Master 127.0.198.62:42329 requested a full tablet report, sending...
I20260929 18:39:44.183785 18919 ts_manager.cc:194] Registered new tserver with Master: 1acf2831daeb433a9a6432588e8fbc5d (127.0.198.3:42631)
I20260929 18:39:44.184314 18919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:42099
I20260929 18:39:44.190027 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:44.199437 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.191866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58562 (local address 127.0.198.62:42329)
0929 18:39:44.191974 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.191978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.192165 (+ 187us) server_negotiation.cc:407] Connection header received
0929 18:39:44.192265 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.192268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.192312 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.192385 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:44.193067 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.193582 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.194524 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.194764 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.194859 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.196271 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:44.196275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:44.196278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:44.196304 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:44.197732 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:44.198306 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:44.198309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:44.198310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:44.198384 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:44.198750 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:44.198752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:44.198753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:44.198940 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:44.199046 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.199251 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.199325 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:39:44.202399 18919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:44.205396 18919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:44.220027 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.213649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:37597 (local address 127.0.0.1:41734)
0929 18:39:44.214580 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.214630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:44.214765 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:44.215419 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:44.215425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:44.215446 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:44.215617 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.215624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.217210 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.217212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.218422 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.218431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.219411 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.219415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.219568 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.219605 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:39:44.219720 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:39:44.219828 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":90,"thread_start_us":80,"threads_started":1}
I20260929 18:39:44.220065 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.214096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.0.198.3:42631)
0929 18:39:44.214411 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.214422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.214806 (+ 384us) server_negotiation.cc:407] Connection header received
0929 18:39:44.215023 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.215029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.215243 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.215374 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:44.215795 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.216822 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.218749 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.219474 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.219575 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.219699 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.219839 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.219931 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":136,"threads_started":1}
I20260929 18:39:44.220186 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.214396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37915 (local address 127.0.0.1:54344)
0929 18:39:44.214980 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.214993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:44.215062 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:44.215684 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:44.215686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:44.215700 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:44.215782 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.215787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.217183 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.217186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.218632 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.218643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.219430 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.219433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.219942 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.219964 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:44.220028 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:39:44.220087 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":507,"thread_start_us":100,"threads_started":1}
I20260929 18:39:44.220347 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.213781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.0.198.2:37597)
0929 18:39:44.214239 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.214251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.214970 (+ 719us) server_negotiation.cc:407] Connection header received
0929 18:39:44.215054 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.215059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.215256 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.215749 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:44.215912 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.217046 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.218530 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.219288 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.219399 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.219551 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.220119 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.220214 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":123,"threads_started":1}
I20260929 18:39:44.221395 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.214544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.198.1:37915)
0929 18:39:44.215291 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.215305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.215326 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:39:44.215394 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.215398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.215557 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.215705 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:44.215854 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.217046 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.218770 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.219313 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.219559 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.219696 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.221183 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.221276 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":108,"threads_started":1}
I20260929 18:39:44.220031 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.213732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:42631 (local address 127.0.0.1:45778)
0929 18:39:44.214667 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.214702 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:44.214878 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:44.215565 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:44.215568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:44.215585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:44.215674 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.215679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.217039 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.217042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.218632 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.218645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.219624 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.219628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.219712 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.219734 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:44.219801 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:39:44.219871 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":124,"thread_start_us":129,"threads_started":1}
I20260929 18:39:44.223299 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 88f2abbcb8e3444e82840ad04a3a0da1 (DEFAULT_TABLE table=test-table [id=621384c411314733b5836368f94a1886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:44.224269 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f2abbcb8e3444e82840ad04a3a0da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:44.223904 19044 tablet_service.cc:1467] Processing CreateTablet for tablet 88f2abbcb8e3444e82840ad04a3a0da1 (DEFAULT_TABLE table=test-table [id=621384c411314733b5836368f94a1886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:44.224889 19044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f2abbcb8e3444e82840ad04a3a0da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:44.226421 19312 tablet_service.cc:1467] Processing CreateTablet for tablet 88f2abbcb8e3444e82840ad04a3a0da1 (DEFAULT_TABLE table=test-table [id=621384c411314733b5836368f94a1886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:44.227392 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f2abbcb8e3444e82840ad04a3a0da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:44.231889 19396 tablet_bootstrap.cc:492] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Bootstrap starting.
I20260929 18:39:44.232406 19395 tablet_bootstrap.cc:492] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e: Bootstrap starting.
I20260929 18:39:44.234266 19395 tablet_bootstrap.cc:654] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:44.234606 19396 tablet_bootstrap.cc:654] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:44.234790 19397 tablet_bootstrap.cc:492] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Bootstrap starting.
I20260929 18:39:44.235414 19396 log.cc:824] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:44.235430 19395 log.cc:824] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:44.236848 19397 tablet_bootstrap.cc:654] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:44.237387 19396 tablet_bootstrap.cc:492] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: No bootstrap required, opened a new log
I20260929 18:39:44.237550 19395 tablet_bootstrap.cc:492] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e: No bootstrap required, opened a new log
I20260929 18:39:44.237584 19396 ts_tablet_manager.cc:1397] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:39:44.237704 19395 ts_tablet_manager.cc:1397] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:44.237864 19397 log.cc:824] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:44.239363 19397 tablet_bootstrap.cc:492] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: No bootstrap required, opened a new log
I20260929 18:39:44.239531 19397 ts_tablet_manager.cc:1397] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:44.241420 19395 raft_consensus.cc:348] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.241420 19396 raft_consensus.cc:348] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.241694 19396 raft_consensus.cc:374] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:44.241694 19395 raft_consensus.cc:374] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:44.241751 19395 raft_consensus.cc:729] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f49d95bc4c1411badff79dd28b74f1e, State: Initialized, Role: FOLLOWER
I20260929 18:39:44.241751 19396 raft_consensus.cc:729] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dcb3b32200b4d619d4cfaf154a7e543, State: Initialized, Role: FOLLOWER
I20260929 18:39:44.242229 19396 consensus_queue.cc:260] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.242229 19395 consensus_queue.cc:260] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.242360 19397 raft_consensus.cc:348] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.242660 19397 raft_consensus.cc:374] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:44.242718 19397 raft_consensus.cc:729] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acf2831daeb433a9a6432588e8fbc5d, State: Initialized, Role: FOLLOWER
I20260929 18:39:44.243045 19244 heartbeater.cc:503] Master 127.0.198.62:42329 was elected leader, sending a full tablet report...
I20260929 18:39:44.243187 19396 ts_tablet_manager.cc:1428] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:39:44.243129 19397 consensus_queue.cc:260] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.243927 19378 heartbeater.cc:503] Master 127.0.198.62:42329 was elected leader, sending a full tablet report...
I20260929 18:39:44.244056 19397 ts_tablet_manager.cc:1428] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 18:39:44.244616 19395 ts_tablet_manager.cc:1428] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:39:44.244940 19110 heartbeater.cc:503] Master 127.0.198.62:42329 was elected leader, sending a full tablet report...
I20260929 18:39:44.306818 19401 raft_consensus.cc:482] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:44.307040 19401 raft_consensus.cc:504] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.308233 19401 leader_election.cc:288] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dcb3b32200b4d619d4cfaf154a7e543 (127.0.198.2:37597), 1acf2831daeb433a9a6432588e8fbc5d (127.0.198.3:42631)
I20260929 18:39:44.313187 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.308613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:37145 (local address 127.0.198.2:37597)
0929 18:39:44.308742 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.308746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.309026 (+ 280us) server_negotiation.cc:407] Connection header received
0929 18:39:44.309143 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.309146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.309184 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.309247 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:44.309555 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.310093 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.311886 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.312531 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.312657 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.312722 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.312984 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.313054 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:39:44.313222 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.308652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:53871 (local address 127.0.198.3:42631)
0929 18:39:44.308789 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.308793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.309002 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:39:44.309126 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.309130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.309177 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.309248 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:44.309562 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.310105 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.311886 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.312571 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.312664 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.312722 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.312984 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.313054 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:39:44.313254 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.308546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:37597 (local address 127.0.198.1:37145)
0929 18:39:44.308923 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.308934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:44.309043 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:44.309272 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:44.309275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:44.309317 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:44.309449 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.309454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.310213 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.310216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.311734 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.311745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.312705 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.312709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.312842 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.312894 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:39:44.312962 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:39:44.313071 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":181,"spinlock_wait_cycles":2176,"thread_start_us":134,"threads_started":1}
I20260929 18:39:44.313261 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.308544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:42631 (local address 127.0.198.1:53871)
0929 18:39:44.308901 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.308913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:44.309022 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:44.309272 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:44.309274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:44.309315 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:44.309454 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.309459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.310213 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.310216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.311734 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.311745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.312705 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.312709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.312846 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.312883 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:39:44.312962 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:39:44.313075 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":177,"spinlock_wait_cycles":34432,"thread_start_us":164,"threads_started":1}
I20260929 18:39:44.313956 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2abbcb8e3444e82840ad04a3a0da1" candidate_uuid: "3f49d95bc4c1411badff79dd28b74f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" is_pre_election: true
I20260929 18:39:44.314010 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2abbcb8e3444e82840ad04a3a0da1" candidate_uuid: "3f49d95bc4c1411badff79dd28b74f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acf2831daeb433a9a6432588e8fbc5d" is_pre_election: true
I20260929 18:39:44.314337 19198 raft_consensus.cc:2454] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f49d95bc4c1411badff79dd28b74f1e in term 0.
I20260929 18:39:44.314337 19332 raft_consensus.cc:2454] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f49d95bc4c1411badff79dd28b74f1e in term 0.
I20260929 18:39:44.314797 18998 leader_election.cc:302] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acf2831daeb433a9a6432588e8fbc5d, 3f49d95bc4c1411badff79dd28b74f1e; no voters:
I20260929 18:39:44.315071 19401 raft_consensus.cc:2781] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:44.315249 19401 raft_consensus.cc:482] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:44.315299 19401 raft_consensus.cc:3037] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:44.316326 19401 raft_consensus.cc:504] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.316694 19401 leader_election.cc:288] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [CANDIDATE]: Term 1 election: Requested vote from peers 4dcb3b32200b4d619d4cfaf154a7e543 (127.0.198.2:37597), 1acf2831daeb433a9a6432588e8fbc5d (127.0.198.3:42631)
I20260929 18:39:44.317049 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2abbcb8e3444e82840ad04a3a0da1" candidate_uuid: "3f49d95bc4c1411badff79dd28b74f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dcb3b32200b4d619d4cfaf154a7e543"
I20260929 18:39:44.317072 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2abbcb8e3444e82840ad04a3a0da1" candidate_uuid: "3f49d95bc4c1411badff79dd28b74f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acf2831daeb433a9a6432588e8fbc5d"
I20260929 18:39:44.317171 19198 raft_consensus.cc:3037] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:44.317203 19332 raft_consensus.cc:3037] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:44.318213 19198 raft_consensus.cc:2454] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f49d95bc4c1411badff79dd28b74f1e in term 1.
I20260929 18:39:44.318274 19332 raft_consensus.cc:2454] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f49d95bc4c1411badff79dd28b74f1e in term 1.
I20260929 18:39:44.318599 18998 leader_election.cc:302] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acf2831daeb433a9a6432588e8fbc5d, 3f49d95bc4c1411badff79dd28b74f1e; no voters:
I20260929 18:39:44.318749 19401 raft_consensus.cc:2781] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:44.318951 19401 raft_consensus.cc:686] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [term 1 LEADER]: Becoming Leader. State: Replica: 3f49d95bc4c1411badff79dd28b74f1e, State: Running, Role: LEADER
I20260929 18:39:44.319263 19401 consensus_queue.cc:237] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } }
I20260929 18:39:44.322579 18918 catalog_manager.cc:5495] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f49d95bc4c1411badff79dd28b74f1e (127.0.198.1). New cstate: current_term: 1 leader_uuid: "3f49d95bc4c1411badff79dd28b74f1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f49d95bc4c1411badff79dd28b74f1e" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37915 } health_report { overall_health: HEALTHY } } }
I20260929 18:39:44.344151 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.340436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.0.198.1:37915)
0929 18:39:44.340559 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.340564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.340578 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:39:44.340687 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.340690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.340758 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.340848 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:44.341095 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.341597 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.342654 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.342875 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.342962 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.343302 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:44.343750 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:44.343881 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.343975 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.344012 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:39:44.350206 19332 raft_consensus.cc:1261] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 1 FOLLOWER]: Refusing update from remote peer 3f49d95bc4c1411badff79dd28b74f1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:44.350206 19198 raft_consensus.cc:1261] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 1 FOLLOWER]: Refusing update from remote peer 3f49d95bc4c1411badff79dd28b74f1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:44.350929 19406 consensus_queue.cc:1035] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dcb3b32200b4d619d4cfaf154a7e543" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:44.351137 19401 consensus_queue.cc:1035] T 88f2abbcb8e3444e82840ad04a3a0da1 P 3f49d95bc4c1411badff79dd28b74f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acf2831daeb433a9a6432588e8fbc5d" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:44.358237 19411 mvcc.cc:205] Tried to move back new op lower bound from 7334736627091046400 to 7334736626974261248. Current Snapshot: MvccSnapshot[applied={T|T < 7334736627091046400}]
I20260929 18:39:44.358587 19409 mvcc.cc:205] Tried to move back new op lower bound from 7334736627091046400 to 7334736626974261248. Current Snapshot: MvccSnapshot[applied={T|T < 7334736627091046400}]
I20260929 18:39:44.358906 19410 mvcc.cc:205] Tried to move back new op lower bound from 7334736627091046400 to 7334736626974261248. Current Snapshot: MvccSnapshot[applied={T|T < 7334736627091046400}]
I20260929 18:39:44.366643 18918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:44.366796 18918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:44.369112 18918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9391cc2847e54b3ca94eba56c37ddb05: Sending DeleteTablet for 3 replicas of tablet 88f2abbcb8e3444e82840ad04a3a0da1
I20260929 18:39:44.369894 19178 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f2abbcb8e3444e82840ad04a3a0da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:44 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:41734
I20260929 18:39:44.369939 19312 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f2abbcb8e3444e82840ad04a3a0da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:44 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:45778
I20260929 18:39:44.370111 19044 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f2abbcb8e3444e82840ad04a3a0da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:44 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:54344
I20260929 18:39:44.370564 19419 tablet_replica.cc:330] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: stopping tablet replica
I20260929 18:39:44.370600 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18981
I20260929 18:39:44.370709 19418 tablet_replica.cc:330] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: stopping tablet replica
I20260929 18:39:44.370883 19419 raft_consensus.cc:2229] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:44.370951 19418 raft_consensus.cc:2229] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:44.371183 19419 raft_consensus.cc:2258] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:44.371204 19418 raft_consensus.cc:2258] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:44.372570 19418 ts_tablet_manager.cc:1905] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:44.373337 19419 ts_tablet_manager.cc:1905] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:44.375211 19418 ts_tablet_manager.cc:1918] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:44.375301 19418 log.cc:1196] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/88f2abbcb8e3444e82840ad04a3a0da1
I20260929 18:39:44.375469 19419 ts_tablet_manager.cc:1918] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:44.375552 19419 log.cc:1196] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/88f2abbcb8e3444e82840ad04a3a0da1
I20260929 18:39:44.375586 19418 ts_tablet_manager.cc:1939] T 88f2abbcb8e3444e82840ad04a3a0da1 P 1acf2831daeb433a9a6432588e8fbc5d: Deleting consensus metadata
I20260929 18:39:44.375871 19419 ts_tablet_manager.cc:1939] T 88f2abbcb8e3444e82840ad04a3a0da1 P 4dcb3b32200b4d619d4cfaf154a7e543: Deleting consensus metadata
I20260929 18:39:44.376832 18904 catalog_manager.cc:4841] TS 1acf2831daeb433a9a6432588e8fbc5d (127.0.198.3:42631): tablet 88f2abbcb8e3444e82840ad04a3a0da1 (table test-table [id=621384c411314733b5836368f94a1886]) successfully deleted
I20260929 18:39:44.376935 18905 catalog_manager.cc:4841] TS 4dcb3b32200b4d619d4cfaf154a7e543 (127.0.198.2:37597): tablet 88f2abbcb8e3444e82840ad04a3a0da1 (table test-table [id=621384c411314733b5836368f94a1886]) successfully deleted
W20260929 18:39:44.377776 18904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:37915 (error 108)
I20260929 18:39:44.378463 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 19115
I20260929 18:39:44.378732 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.378461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37915 (local address 127.0.0.1:54376)
0929 18:39:44.378589 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.378655 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:39:44.379859 18904 catalog_manager.cc:4584] TS 3f49d95bc4c1411badff79dd28b74f1e (127.0.198.1:37915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88f2abbcb8e3444e82840ad04a3a0da1: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37915: connect: Connection refused (error 111)
I20260929 18:39:44.383308 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 19249
I20260929 18:39:44.388015 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 18887
2026-09-29T18:39:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T18:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:44Z Disabled control of system clock
I20260929 18:39:44.415534 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:34283
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:39:44.524101 19425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:44.524372 19425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:44.524406 19425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:44.528482 19425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:44.528551 19425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:44.528565 19425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:44.528580 19425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:44.528594 19425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:44.533432 19425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:34283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:44.534593 19425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:44.535948 19425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:39:44.544472 19425 server_base.cc:1034] running on GCE node
W20260929 18:39:44.544473 19431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:44.544440 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:44.544440 19433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:44.545158 19425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:44.546281 19425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:44.547436 19425 hybrid_clock.cc:648] HybridClock initialized: now 1790707184547397 us; error 47 us; skew 500 ppm
I20260929 18:39:44.549441 19425 webserver.cc:460] Webserver started at http://127.0.198.62:38651/ using document root <none> and password file <none>
I20260929 18:39:44.550063 19425 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:44.550141 19425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:44.550357 19425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:44.552403 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "a32be48a4f48cd7274fbfffe4e0c4dde"
server_key_iv: "411e39245b1df7399b464f0160356fc0"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.552938 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "a32be48a4f48cd7274fbfffe4e0c4dde"
server_key_iv: "411e39245b1df7399b464f0160356fc0"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.556612 19425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:39:44.559032 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.560235 19425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:44.560333 19425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "a32be48a4f48cd7274fbfffe4e0c4dde"
server_key_iv: "411e39245b1df7399b464f0160356fc0"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.560439 19425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:44.591977 19425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:44.592733 19425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:44.592913 19425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:44.599797 19425 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:34283
I20260929 18:39:44.599818 19491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:34283 every 8 connection(s)
I20260929 18:39:44.600728 19425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:44.601249 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 19425
I20260929 18:39:44.601369 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:44.601639 792 external_mini_cluster.cc:1428] Setting key 8901cea06562e7585ed1d5d4642667f4
I20260929 18:39:44.603982 19492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:44.609400 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd: Bootstrap starting.
I20260929 18:39:44.610664 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.602938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.0.198.62:34283)
0929 18:39:44.603691 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.603708 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.603742 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:39:44.604411 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.604429 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.604893 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.605213 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:44.605624 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.606618 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.607466 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.607971 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.608111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.608959 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:44.608963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:44.608975 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:44.609347 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:44.609400 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:44.609418 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:44.609493 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.609693 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.609950 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":107,"threads_started":1}
I20260929 18:39:44.611778 19492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:44.612465 19492 log.cc:824] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:44.614346 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd: No bootstrap required, opened a new log
I20260929 18:39:44.617710 19492 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34283 } }
I20260929 18:39:44.617925 19492 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:44.617995 19492 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ce1427d7aa4912872d54b9eb5ff0dd, State: Initialized, Role: FOLLOWER
I20260929 18:39:44.618397 19492 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34283 } }
I20260929 18:39:44.618523 19492 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:44.618584 19492 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:44.618688 19492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:44.619521 19492 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34283 } }
I20260929 18:39:44.619837 19492 leader_election.cc:302] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2ce1427d7aa4912872d54b9eb5ff0dd; no voters:
I20260929 18:39:44.620123 19492 leader_election.cc:288] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:44.620323 19497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:44.620519 19497 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [term 1 LEADER]: Becoming Leader. State: Replica: e2ce1427d7aa4912872d54b9eb5ff0dd, State: Running, Role: LEADER
I20260929 18:39:44.620851 19497 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34283 } }
I20260929 18:39:44.621249 19492 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:44.622306 19499 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ce1427d7aa4912872d54b9eb5ff0dd. Latest consensus state: current_term: 1 leader_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34283 } } }
I20260929 18:39:44.622412 19499 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:44.622609 19498 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ce1427d7aa4912872d54b9eb5ff0dd" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34283 } } }
I20260929 18:39:44.622771 19498 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:44.622841 19506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:44.626106 19506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:44.632210 19506 catalog_manager.cc:1347] Generated new cluster ID: 2f340cdef8094a6ca87fc4e2e2902668
I20260929 18:39:44.632347 19506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:44.642315 19506 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:44.643260 19506 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:44.652571 19506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd: Generated new TSK 0
I20260929 18:39:44.653286 19506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:44.660840 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34283
--builtin_ntp_servers=127.0.198.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:39:44.775780 19516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:44.776054 19516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:44.776094 19516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:44.780131 19516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:44.780187 19516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:44.780241 19516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:44.785015 19516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:44.786160 19516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:44.787534 19516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:39:44.796995 19516 server_base.cc:1034] running on GCE node
W20260929 18:39:44.796939 19521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:44.797043 19522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:44.796936 19524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:44.797525 19516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:44.798228 19516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:44.799388 19516 hybrid_clock.cc:648] HybridClock initialized: now 1790707184799363 us; error 39 us; skew 500 ppm
I20260929 18:39:44.801162 19516 webserver.cc:460] Webserver started at http://127.0.198.1:39071/ using document root <none> and password file <none>
I20260929 18:39:44.801709 19516 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:44.801771 19516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:44.801956 19516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:44.804031 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "a26331a0beb048239fd13b10c26c5553"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "ae7cae20237aeefb2355a2d4da1e27f6"
server_key_iv: "6cbcbc357540d4caf7e2406e46df9bcb"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.804610 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "a26331a0beb048239fd13b10c26c5553"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "ae7cae20237aeefb2355a2d4da1e27f6"
server_key_iv: "6cbcbc357540d4caf7e2406e46df9bcb"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.808358 19516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 18:39:44.810878 19530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.812206 19516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:39:44.812305 19516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "a26331a0beb048239fd13b10c26c5553"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "ae7cae20237aeefb2355a2d4da1e27f6"
server_key_iv: "6cbcbc357540d4caf7e2406e46df9bcb"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.812420 19516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:44.826023 19516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:44.827026 19516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:44.827251 19516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:44.827893 19516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:44.828843 19516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:44.828895 19516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.828935 19516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:44.828955 19516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.837270 19516 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:38815
I20260929 18:39:44.837297 19643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:38815 every 8 connection(s)
I20260929 18:39:44.838248 19516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:44.847344 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.840005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:48917 (local address 127.0.198.62:34283)
0929 18:39:44.840199 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:44.840202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:44.841341 (+ 1139us) server_negotiation.cc:407] Connection header received
0929 18:39:44.842743 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:44.842746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:44.842802 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:44.842873 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:44.843817 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.844349 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.845492 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.845725 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.845807 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.846880 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:44.846883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:44.846886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:44.846898 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:44.846917 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:44.846925 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:44.846976 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:44.847142 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 18:39:44.847226 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:39:44.847883 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 19516
I20260929 18:39:44.848007 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:44.848284 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:44.840465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34283 (local address 127.0.198.1:48917)
0929 18:39:44.841185 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:44.841230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:44.842537 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:44.843066 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:44.843073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:44.843168 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:44.843679 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.843693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.844485 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.844488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.845373 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:44.845382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:44.845889 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:44.845893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:44.846077 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:44.846621 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:44.846623 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:44.846664 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:44.846734 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:44.846987 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:44.846992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:44.846996 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:39:44.847090 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:39:44.847324 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260929 18:39:44.848318 792 external_mini_cluster.cc:1428] Setting key 8456840a0950c4d1097f88fef0340ddc
I20260929 18:39:44.849594 19644 heartbeater.cc:347] Connected to a master server at 127.0.198.62:34283
I20260929 18:39:44.849833 19644 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:44.850239 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34283
--builtin_ntp_servers=127.0.198.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:39:44.850363 19644 heartbeater.cc:511] Master 127.0.198.62:34283 requested a full tablet report, sending...
I20260929 18:39:44.852061 19455 ts_manager.cc:194] Registered new tserver with Master: a26331a0beb048239fd13b10c26c5553 (127.0.198.1:38815)
I20260929 18:39:44.853614 19455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.1:48917
W20260929 18:39:44.960472 19647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:44.960757 19647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:44.960796 19647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:44.964924 19647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:44.964986 19647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:44.965044 19647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:44.969877 19647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:44.970937 19647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:44.972251 19647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:44.981567 19653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:44.981564 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:44.981619 19647 server_base.cc:1034] running on GCE node
W20260929 18:39:44.981571 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:44.982266 19647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:44.982890 19647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:44.984050 19647 hybrid_clock.cc:648] HybridClock initialized: now 1790707184984038 us; error 28 us; skew 500 ppm
I20260929 18:39:44.985918 19647 webserver.cc:460] Webserver started at http://127.0.198.2:40357/ using document root <none> and password file <none>
I20260929 18:39:44.986519 19647 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:44.986581 19647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:44.986769 19647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:44.988832 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "e55554bd1e7368dafcf0641a263e3a1c"
server_key_iv: "523e9bef01df319ad716e4f907b746c5"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.989372 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "e55554bd1e7368dafcf0641a263e3a1c"
server_key_iv: "523e9bef01df319ad716e4f907b746c5"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.993108 19647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:39:44.995694 19661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:44.996933 19647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:39:44.997045 19647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7"
format_stamp: "Formatted at 2026-09-29 18:39:44 on dist-test-slave-q5qx"
server_key: "e55554bd1e7368dafcf0641a263e3a1c"
server_key_iv: "523e9bef01df319ad716e4f907b746c5"
server_key_version: "encryptionkey@0"
I20260929 18:39:44.997146 19647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:45.010258 19647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:45.010938 19647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:45.011103 19647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:45.011709 19647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:45.012619 19647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:45.012665 19647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:45.012717 19647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:45.012746 19647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:45.021130 19647 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:37647
I20260929 18:39:45.021171 19774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:37647 every 8 connection(s)
I20260929 18:39:45.022126 19647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:45.027520 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 19647
I20260929 18:39:45.027649 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:39:45.027953 792 external_mini_cluster.cc:1428] Setting key cf7f7e97345942f0d6da4e300c141036
I20260929 18:39:45.030160 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34283
--builtin_ntp_servers=127.0.198.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:39:45.032570 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.024039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:52423 (local address 127.0.198.62:34283)
0929 18:39:45.024234 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.024237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.025439 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 18:39:45.026898 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.026901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.026951 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.027025 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:45.028296 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.028946 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.030501 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.030726 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.030808 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.032046 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:45.032050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:45.032053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:45.032067 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:45.032088 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:45.032097 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:45.032160 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.032357 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.032428 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 18:39:45.033481 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.024502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34283 (local address 127.0.198.2:52423)
0929 18:39:45.025273 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.025319 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.026679 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.027259 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.027270 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.027384 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:45.028109 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.028130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.029073 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.029076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.030383 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.030395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.030874 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.030878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.031081 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.031776 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:45.031778 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:45.031821 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:45.031893 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:45.032216 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:45.032224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:45.032229 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.032348 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.032657 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260929 18:39:45.034636 19775 heartbeater.cc:347] Connected to a master server at 127.0.198.62:34283
I20260929 18:39:45.034873 19775 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:45.035416 19775 heartbeater.cc:511] Master 127.0.198.62:34283 requested a full tablet report, sending...
I20260929 18:39:45.036473 19455 ts_manager.cc:194] Registered new tserver with Master: 709a8afd6d4249d4a3c7d9e5fb1e77a7 (127.0.198.2:37647)
I20260929 18:39:45.037253 19455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.2:52423
W20260929 18:39:45.141036 19778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:45.141316 19778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:45.141367 19778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:45.145540 19778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:45.145606 19778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:45.145663 19778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:45.150507 19778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:45.151567 19778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:45.153040 19778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:45.162324 19786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:45.162312 19783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:45.162317 19784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:45.162508 19778 server_base.cc:1034] running on GCE node
I20260929 18:39:45.162886 19778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:45.163513 19778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:45.164686 19778 hybrid_clock.cc:648] HybridClock initialized: now 1790707185164672 us; error 34 us; skew 500 ppm
I20260929 18:39:45.166512 19778 webserver.cc:460] Webserver started at http://127.0.198.3:38647/ using document root <none> and password file <none>
I20260929 18:39:45.167076 19778 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:45.167145 19778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:45.167328 19778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:45.169409 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "4350f3a5b3254760a92d1f890bd94452"
format_stamp: "Formatted at 2026-09-29 18:39:45 on dist-test-slave-q5qx"
server_key: "696cfa29a84e1a9861570a8bea659e91"
server_key_iv: "392538b4ed6fcc101afb63014228d279"
server_key_version: "encryptionkey@0"
I20260929 18:39:45.169940 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "4350f3a5b3254760a92d1f890bd94452"
format_stamp: "Formatted at 2026-09-29 18:39:45 on dist-test-slave-q5qx"
server_key: "696cfa29a84e1a9861570a8bea659e91"
server_key_iv: "392538b4ed6fcc101afb63014228d279"
server_key_version: "encryptionkey@0"
I20260929 18:39:45.173485 19778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:45.175973 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:45.177211 19778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:45.177318 19778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "4350f3a5b3254760a92d1f890bd94452"
format_stamp: "Formatted at 2026-09-29 18:39:45 on dist-test-slave-q5qx"
server_key: "696cfa29a84e1a9861570a8bea659e91"
server_key_iv: "392538b4ed6fcc101afb63014228d279"
server_key_version: "encryptionkey@0"
I20260929 18:39:45.177414 19778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:45.198216 19778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:45.198899 19778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:45.199061 19778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:45.199635 19778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:45.200649 19778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:45.200695 19778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:45.200747 19778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:45.200774 19778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:45.209352 19778 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:44129
I20260929 18:39:45.209376 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:44129 every 8 connection(s)
I20260929 18:39:45.210276 19778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:45.216581 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 19778
I20260929 18:39:45.216681 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:45.216982 792 external_mini_cluster.cc:1428] Setting key 4346d003826430b24b7d20a1c04fb4bb
I20260929 18:39:45.219787 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.212042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:44441 (local address 127.0.198.62:34283)
0929 18:39:45.212209 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.212214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.213243 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 18:39:45.214647 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.214650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.214700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.214786 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:45.215703 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.216342 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.217703 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.217959 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.218084 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.219326 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:45.219330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:45.219333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:45.219344 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:45.219362 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:45.219370 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:45.219426 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.219572 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.219632 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:39:45.220607 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.212453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34283 (local address 127.0.198.3:44441)
0929 18:39:45.213095 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.213136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.214433 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.214947 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.214954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.215040 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:45.215534 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.215547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.216475 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.216478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.217608 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.217618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.218114 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.218118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.218311 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.219024 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:45.219026 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:45.219083 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:45.219187 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:45.219447 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:45.219455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:45.219460 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.219566 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.219843 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260929 18:39:45.221686 19906 heartbeater.cc:347] Connected to a master server at 127.0.198.62:34283
I20260929 18:39:45.221915 19906 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:45.222476 19906 heartbeater.cc:511] Master 127.0.198.62:34283 requested a full tablet report, sending...
I20260929 18:39:45.223487 19455 ts_manager.cc:194] Registered new tserver with Master: 4350f3a5b3254760a92d1f890bd94452 (127.0.198.3:44129)
I20260929 18:39:45.224045 19455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.3:44441
I20260929 18:39:45.231159 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:45.236770 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.233114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.0.198.62:34283)
0929 18:39:45.233283 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.233288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.233402 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:39:45.233498 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.233500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.233540 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.233602 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:45.233848 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.234396 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.235176 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.235427 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.235509 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.236295 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:45.236299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:45.236301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:45.236313 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:45.236333 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:45.236342 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:45.236456 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.236569 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.236607 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 18:39:45.239558 19455 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:45.242496 19455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:45.256722 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.251257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.0.198.2:37647)
0929 18:39:45.252472 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.252484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.252508 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:39:45.252603 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.252607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.252777 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.252931 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:45.253130 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.254227 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.255486 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.256022 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.256151 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.256350 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.256452 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.256547 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":432,"threads_started":1}
I20260929 18:39:45.256717 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.251049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:37647 (local address 127.0.0.1:56100)
0929 18:39:45.251745 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.251764 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.251839 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.252917 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.252919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.252937 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:45.253027 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.253033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.254347 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.254350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.255382 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.255390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.256161 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.256164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.256283 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.256321 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.256427 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.256547 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260929 18:39:45.256753 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.250897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38815 (local address 127.0.0.1:33344)
0929 18:39:45.251329 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.251376 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.251495 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.252580 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.252586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.252610 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:45.252800 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.252809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.253933 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.253936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.255264 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.255275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.256332 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.256336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.256456 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.256478 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.256537 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.256608 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":19,"thread_start_us":72,"threads_started":1}
I20260929 18:39:45.257267 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.251623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:44129 (local address 127.0.0.1:33456)
0929 18:39:45.251966 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.251978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.252047 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.252688 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.252691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.252710 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:45.252848 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.252856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.254314 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.254317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.255417 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.255425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.256361 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.256364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.256998 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.257023 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.257089 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.257156 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":540,"thread_start_us":98,"threads_started":1}
I20260929 18:39:45.257517 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.251580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.0.198.1:38815)
0929 18:39:45.252071 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.252083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.252108 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:39:45.252188 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.252193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.252409 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.252947 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:45.253074 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.253823 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.255402 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.256192 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.257046 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.257235 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.257282 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.257376 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":124,"threads_started":1}
I20260929 18:39:45.257663 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.252014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.0.198.3:44129)
0929 18:39:45.252305 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.252315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.252335 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:39:45.252398 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.252402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.252560 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.252705 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:45.252968 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.254191 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.255533 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.256255 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.256452 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.256641 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.257413 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.257505 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
I20260929 18:39:45.259938 19840 tablet_service.cc:1467] Processing CreateTablet for tablet 6546ae3ad0694d9295da009c903d9090 (DEFAULT_TABLE table=test-table [id=5927311902b14fbf99308fbe35d6a87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:45.259950 19709 tablet_service.cc:1467] Processing CreateTablet for tablet 6546ae3ad0694d9295da009c903d9090 (DEFAULT_TABLE table=test-table [id=5927311902b14fbf99308fbe35d6a87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:45.259950 19578 tablet_service.cc:1467] Processing CreateTablet for tablet 6546ae3ad0694d9295da009c903d9090 (DEFAULT_TABLE table=test-table [id=5927311902b14fbf99308fbe35d6a87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:45.260865 19840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6546ae3ad0694d9295da009c903d9090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:45.260936 19709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6546ae3ad0694d9295da009c903d9090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:45.260936 19578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6546ae3ad0694d9295da009c903d9090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:45.267463 19923 tablet_bootstrap.cc:492] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Bootstrap starting.
I20260929 18:39:45.268594 19925 tablet_bootstrap.cc:492] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553: Bootstrap starting.
I20260929 18:39:45.268594 19924 tablet_bootstrap.cc:492] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Bootstrap starting.
I20260929 18:39:45.269506 19923 tablet_bootstrap.cc:654] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:45.270249 19923 log.cc:824] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:45.270750 19925 tablet_bootstrap.cc:654] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:45.270751 19924 tablet_bootstrap.cc:654] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:45.271552 19925 log.cc:824] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:45.271552 19924 log.cc:824] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:45.271886 19923 tablet_bootstrap.cc:492] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: No bootstrap required, opened a new log
I20260929 18:39:45.272068 19923 ts_tablet_manager.cc:1397] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:45.273504 19924 tablet_bootstrap.cc:492] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: No bootstrap required, opened a new log
I20260929 18:39:45.273558 19925 tablet_bootstrap.cc:492] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553: No bootstrap required, opened a new log
I20260929 18:39:45.273682 19924 ts_tablet_manager.cc:1397] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:45.273721 19925 ts_tablet_manager.cc:1397] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:45.276299 19923 raft_consensus.cc:348] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.276580 19923 raft_consensus.cc:374] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:45.276638 19923 raft_consensus.cc:729] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4350f3a5b3254760a92d1f890bd94452, State: Initialized, Role: FOLLOWER
I20260929 18:39:45.276824 19925 raft_consensus.cc:348] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.277048 19925 raft_consensus.cc:374] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:45.277100 19925 raft_consensus.cc:729] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26331a0beb048239fd13b10c26c5553, State: Initialized, Role: FOLLOWER
I20260929 18:39:45.277081 19923 consensus_queue.cc:260] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.277515 19925 consensus_queue.cc:260] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.277740 19906 heartbeater.cc:503] Master 127.0.198.62:34283 was elected leader, sending a full tablet report...
I20260929 18:39:45.277595 19924 raft_consensus.cc:348] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.277814 19924 raft_consensus.cc:374] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:45.277858 19924 raft_consensus.cc:729] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709a8afd6d4249d4a3c7d9e5fb1e77a7, State: Initialized, Role: FOLLOWER
I20260929 18:39:45.278139 19923 ts_tablet_manager.cc:1428] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:39:45.278309 19924 consensus_queue.cc:260] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.278430 19925 ts_tablet_manager.cc:1428] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:39:45.278733 19644 heartbeater.cc:503] Master 127.0.198.62:34283 was elected leader, sending a full tablet report...
I20260929 18:39:45.278899 19775 heartbeater.cc:503] Master 127.0.198.62:34283 was elected leader, sending a full tablet report...
I20260929 18:39:45.280615 19924 ts_tablet_manager.cc:1428] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 18:39:45.339464 19645 tablet.cc:2369] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:45.405778 19929 raft_consensus.cc:482] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:45.406008 19929 raft_consensus.cc:504] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.407181 19929 leader_election.cc:288] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26331a0beb048239fd13b10c26c5553 (127.0.198.1:38815), 709a8afd6d4249d4a3c7d9e5fb1e77a7 (127.0.198.2:37647)
I20260929 18:39:45.409082 19930 raft_consensus.cc:482] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:45.409276 19930 raft_consensus.cc:504] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.410557 19930 leader_election.cc:288] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 709a8afd6d4249d4a3c7d9e5fb1e77a7 (127.0.198.2:37647), 4350f3a5b3254760a92d1f890bd94452 (127.0.198.3:44129)
I20260929 18:39:45.411226 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.407371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38815 (local address 127.0.198.3:35997)
0929 18:39:45.407509 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.407524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.407619 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.407965 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.407968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.407986 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:45.408105 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.408111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.408823 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.408826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.409917 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.409929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.410800 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.410804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.410906 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.410929 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.411000 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.411076 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"spinlock_wait_cycles":8064}
I20260929 18:39:45.411687 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.407555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:53181 (local address 127.0.198.2:37647)
0929 18:39:45.407717 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.407722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.407994 (+ 272us) server_negotiation.cc:407] Connection header received
0929 18:39:45.408115 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.408119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.408170 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.408231 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:45.408477 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.409051 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.410207 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.410748 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.410847 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.410903 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.411487 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.411572 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:39:45.411834 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.407645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:35997 (local address 127.0.198.1:38815)
0929 18:39:45.407776 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.407780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.407791 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:39:45.407834 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.407836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.407874 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.407936 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:45.408207 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.408708 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.410063 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.410541 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.410639 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.410700 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.411641 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.411717 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:39:45.412058 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.407471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:37647 (local address 127.0.198.3:53181)
0929 18:39:45.407906 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.407916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.408015 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.408246 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.408249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.408263 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:45.408379 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.408384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.409160 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.409164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.410078 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.410088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.410961 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.410965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.411350 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.411372 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.411854 (+ 482us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.411925 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":23,"thread_start_us":132,"threads_started":1}
I20260929 18:39:45.412329 19598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546ae3ad0694d9295da009c903d9090" candidate_uuid: "4350f3a5b3254760a92d1f890bd94452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26331a0beb048239fd13b10c26c5553" is_pre_election: true
I20260929 18:39:45.412542 19598 raft_consensus.cc:2454] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4350f3a5b3254760a92d1f890bd94452 in term 0.
I20260929 18:39:45.413004 19796 leader_election.cc:302] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4350f3a5b3254760a92d1f890bd94452, a26331a0beb048239fd13b10c26c5553; no voters:
I20260929 18:39:45.413280 19929 raft_consensus.cc:2781] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:45.413440 19929 raft_consensus.cc:482] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:45.413482 19929 raft_consensus.cc:3037] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:45.414197 19729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546ae3ad0694d9295da009c903d9090" candidate_uuid: "4350f3a5b3254760a92d1f890bd94452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" is_pre_election: true
I20260929 18:39:45.414482 19729 raft_consensus.cc:2454] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4350f3a5b3254760a92d1f890bd94452 in term 0.
I20260929 18:39:45.414793 19929 raft_consensus.cc:504] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.415031 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.411160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:45873 (local address 127.0.198.3:44129)
0929 18:39:45.411267 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.411271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.411329 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:39:45.411486 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.411489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.411528 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.411567 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:45.411928 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.412453 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.413856 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.414415 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.414481 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.414530 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.414858 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.414925 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:39:45.415177 19929 leader_election.cc:288] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [CANDIDATE]: Term 1 election: Requested vote from peers a26331a0beb048239fd13b10c26c5553 (127.0.198.1:38815), 709a8afd6d4249d4a3c7d9e5fb1e77a7 (127.0.198.2:37647)
I20260929 18:39:45.415324 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.410728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:37647 (local address 127.0.198.1:38945)
0929 18:39:45.410950 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.410964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.411050 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.411914 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.411917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.411935 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:45.412036 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.412042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.413009 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.413012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.414115 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.414125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.414938 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.414941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.415037 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.415060 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.415108 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.415187 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260929 18:39:45.415347 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.411067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:44129 (local address 127.0.198.1:45873)
0929 18:39:45.411263 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.411275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:45.411384 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:45.411689 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:45.411691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:45.411709 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:45.411812 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.411819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.412560 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.412562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.413722 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:45.413732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.414625 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.414629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:45.414726 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.414748 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:45.415186 (+ 438us) client_negotiation.cc:240] Negotiation successful
0929 18:39:45.415243 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260929 18:39:45.415753 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.411370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:38945 (local address 127.0.198.2:37647)
0929 18:39:45.411694 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.411702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.411713 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:39:45.411750 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.411753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.411795 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.412133 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:45.412223 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.412898 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.414247 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.414828 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.415483 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.415536 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.415566 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.415627 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":100,"threads_started":1}
I20260929 18:39:45.415742 19598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546ae3ad0694d9295da009c903d9090" candidate_uuid: "4350f3a5b3254760a92d1f890bd94452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26331a0beb048239fd13b10c26c5553"
I20260929 18:39:45.415810 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546ae3ad0694d9295da009c903d9090" candidate_uuid: "a26331a0beb048239fd13b10c26c5553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4350f3a5b3254760a92d1f890bd94452" is_pre_election: true
I20260929 18:39:45.415859 19598 raft_consensus.cc:3037] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:45.415968 19729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546ae3ad0694d9295da009c903d9090" candidate_uuid: "4350f3a5b3254760a92d1f890bd94452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7"
I20260929 18:39:45.416029 19860 raft_consensus.cc:2379] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a26331a0beb048239fd13b10c26c5553 in current term 1: Already voted for candidate 4350f3a5b3254760a92d1f890bd94452 in this term.
I20260929 18:39:45.416067 19729 raft_consensus.cc:3037] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:45.416317 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546ae3ad0694d9295da009c903d9090" candidate_uuid: "a26331a0beb048239fd13b10c26c5553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" is_pre_election: true
I20260929 18:39:45.416884 19533 leader_election.cc:302] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a26331a0beb048239fd13b10c26c5553; no voters: 4350f3a5b3254760a92d1f890bd94452, 709a8afd6d4249d4a3c7d9e5fb1e77a7
I20260929 18:39:45.417101 19598 raft_consensus.cc:2454] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4350f3a5b3254760a92d1f890bd94452 in term 1.
I20260929 18:39:45.417165 19930 raft_consensus.cc:2726] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:39:45.417326 19729 raft_consensus.cc:2454] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4350f3a5b3254760a92d1f890bd94452 in term 1.
I20260929 18:39:45.417409 19796 leader_election.cc:302] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4350f3a5b3254760a92d1f890bd94452, a26331a0beb048239fd13b10c26c5553; no voters:
I20260929 18:39:45.417569 19929 raft_consensus.cc:2781] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:45.417774 19929 raft_consensus.cc:686] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 1 LEADER]: Becoming Leader. State: Replica: 4350f3a5b3254760a92d1f890bd94452, State: Running, Role: LEADER
I20260929 18:39:45.418115 19929 consensus_queue.cc:237] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } }
I20260929 18:39:45.421473 19450 catalog_manager.cc:5495] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4350f3a5b3254760a92d1f890bd94452 (127.0.198.3). New cstate: current_term: 1 leader_uuid: "4350f3a5b3254760a92d1f890bd94452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4350f3a5b3254760a92d1f890bd94452" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 44129 } health_report { overall_health: HEALTHY } } }
I20260929 18:39:45.447751 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.444213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.0.198.3:44129)
0929 18:39:45.444358 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:45.444363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:45.444376 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:39:45.444444 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:45.444446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:45.444497 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:45.444594 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:45.444847 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.445335 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.446246 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:45.446455 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:45.446549 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:45.446895 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:45.447305 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:45.447456 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:45.447594 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:39:45.447637 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:39:45.453615 19729 raft_consensus.cc:1261] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 1 FOLLOWER]: Refusing update from remote peer 4350f3a5b3254760a92d1f890bd94452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:45.453677 19598 raft_consensus.cc:1261] T 6546ae3ad0694d9295da009c903d9090 P a26331a0beb048239fd13b10c26c5553 [term 1 FOLLOWER]: Refusing update from remote peer 4350f3a5b3254760a92d1f890bd94452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:45.454260 19929 consensus_queue.cc:1035] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "709a8afd6d4249d4a3c7d9e5fb1e77a7" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 37647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:45.454456 19936 consensus_queue.cc:1035] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26331a0beb048239fd13b10c26c5553" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:39:45.461122 19907 tablet.cc:2369] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:45.463644 19941 mvcc.cc:205] Tried to move back new op lower bound from 7334736631610376192 to 7334736631475367936. Current Snapshot: MvccSnapshot[applied={T|T < 7334736631610376192}]
I20260929 18:39:45.463929 19940 mvcc.cc:205] Tried to move back new op lower bound from 7334736631610376192 to 7334736631475367936. Current Snapshot: MvccSnapshot[applied={T|T < 7334736631610376192}]
I20260929 18:39:45.471315 19456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:45.471480 19456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:45.473773 19456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2ce1427d7aa4912872d54b9eb5ff0dd: Sending DeleteTablet for 3 replicas of tablet 6546ae3ad0694d9295da009c903d9090
I20260929 18:39:45.474593 19578 tablet_service.cc:1514] Processing DeleteTablet for tablet 6546ae3ad0694d9295da009c903d9090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:45 UTC) from {username='slave'} at 127.0.0.1:33344
I20260929 18:39:45.474676 19709 tablet_service.cc:1514] Processing DeleteTablet for tablet 6546ae3ad0694d9295da009c903d9090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:45 UTC) from {username='slave'} at 127.0.0.1:56100
I20260929 18:39:45.474754 19840 tablet_service.cc:1514] Processing DeleteTablet for tablet 6546ae3ad0694d9295da009c903d9090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:45 UTC) from {username='slave'} at 127.0.0.1:33456
I20260929 18:39:45.475345 19952 tablet_replica.cc:330] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: stopping tablet replica
I20260929 18:39:45.475637 19952 raft_consensus.cc:2229] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:45.475684 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 19516
I20260929 18:39:45.475939 19952 raft_consensus.cc:2258] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:45.476254 19953 tablet_replica.cc:330] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: stopping tablet replica
I20260929 18:39:45.476526 19953 raft_consensus.cc:2229] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:39:45.476866 19953 raft_consensus.cc:2258] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:45.478289 19952 ts_tablet_manager.cc:1905] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:45.478574 19953 ts_tablet_manager.cc:1905] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:45.480971 19952 ts_tablet_manager.cc:1918] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:45.480971 19953 ts_tablet_manager.cc:1918] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:45.481072 19952 log.cc:1196] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/6546ae3ad0694d9295da009c903d9090
I20260929 18:39:45.481191 19953 log.cc:1196] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/6546ae3ad0694d9295da009c903d9090
I20260929 18:39:45.481361 19952 ts_tablet_manager.cc:1939] T 6546ae3ad0694d9295da009c903d9090 P 709a8afd6d4249d4a3c7d9e5fb1e77a7: Deleting consensus metadata
I20260929 18:39:45.481633 19953 ts_tablet_manager.cc:1939] T 6546ae3ad0694d9295da009c903d9090 P 4350f3a5b3254760a92d1f890bd94452: Deleting consensus metadata
I20260929 18:39:45.482663 19442 catalog_manager.cc:4841] TS 4350f3a5b3254760a92d1f890bd94452 (127.0.198.3:44129): tablet 6546ae3ad0694d9295da009c903d9090 (table test-table [id=5927311902b14fbf99308fbe35d6a87b]) successfully deleted
I20260929 18:39:45.482861 19442 catalog_manager.cc:4841] TS 709a8afd6d4249d4a3c7d9e5fb1e77a7 (127.0.198.2:37647): tablet 6546ae3ad0694d9295da009c903d9090 (table test-table [id=5927311902b14fbf99308fbe35d6a87b]) successfully deleted
W20260929 18:39:45.485782 19443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:38815 (error 108)
I20260929 18:39:45.486567 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 19647
I20260929 18:39:45.486840 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:45.486568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:38815 (local address 127.0.0.1:33352)
0929 18:39:45.486694 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:45.486757 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:38815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:39:45.488504 19443 catalog_manager.cc:4584] TS a26331a0beb048239fd13b10c26c5553 (127.0.198.1:38815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6546ae3ad0694d9295da009c903d9090: Network error: Client connection negotiation failed: client connection to 127.0.198.1:38815: connect: Connection refused (error 111)
I20260929 18:39:45.492220 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 19778
I20260929 18:39:45.497375 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 19425
2026-09-29T18:39:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1109 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:45 dist-test-slave-q5qx krb5kdc[19957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:45 dist-test-slave-q5qx krb5kdc[19957](info): set up 2 sockets
Sep 29 18:39:45 dist-test-slave-q5qx krb5kdc[19957](info): commencing operation
krb5kdc: starting...
W20260929 18:39:46.453795 792 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:46 dist-test-slave-q5qx krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:39:46.577478 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44547
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:46.688396 19970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:46.688665 19970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:46.688706 19970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:46.692691 19970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:46.692747 19970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:46.692765 19970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:46.692781 19970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:46.692794 19970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:46.698091 19970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:46.699074 19970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:46.700006 19970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:46.708256 19978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:46.708325 19975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:46.708271 19976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:46.708515 19970 server_base.cc:1034] running on GCE node
I20260929 18:39:46.709120 19970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:46.710264 19970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:46.711423 19970 hybrid_clock.cc:648] HybridClock initialized: now 1790707186711407 us; error 30 us; skew 500 ppm
Sep 29 18:39:46 dist-test-slave-q5qx krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:46.714048 19970 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:39:46.714941 19970 webserver.cc:460] Webserver started at http://127.0.198.62:35359/ using document root <none> and password file <none>
I20260929 18:39:46.715478 19970 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:46.715536 19970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:46.715745 19970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:46.717787 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "d11c31f6ecd0498c981b5c8e4a59d412"
format_stamp: "Formatted at 2026-09-29 18:39:46 on dist-test-slave-q5qx"
server_key: "32d0510e58e731e787313e1d8f3b4ab4"
server_key_iv: "f577b7df44dd0d27692e2817429c3dd6"
server_key_version: "encryptionkey@0"
I20260929 18:39:46.718364 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "d11c31f6ecd0498c981b5c8e4a59d412"
format_stamp: "Formatted at 2026-09-29 18:39:46 on dist-test-slave-q5qx"
server_key: "32d0510e58e731e787313e1d8f3b4ab4"
server_key_iv: "f577b7df44dd0d27692e2817429c3dd6"
server_key_version: "encryptionkey@0"
I20260929 18:39:46.722075 19970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:46.724349 19985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:46.725540 19970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:46.725652 19970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "d11c31f6ecd0498c981b5c8e4a59d412"
format_stamp: "Formatted at 2026-09-29 18:39:46 on dist-test-slave-q5qx"
server_key: "32d0510e58e731e787313e1d8f3b4ab4"
server_key_iv: "f577b7df44dd0d27692e2817429c3dd6"
server_key_version: "encryptionkey@0"
I20260929 18:39:46.725752 19970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:46.761744 19970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:46.764765 19970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:46.764935 19970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:46.771889 19970 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:44547
I20260929 18:39:46.771919 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:44547 every 8 connection(s)
I20260929 18:39:46.772853 19970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:46.774470 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 19970
I20260929 18:39:46.774629 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:46.775013 792 external_mini_cluster.cc:1428] Setting key 18fa7b2472cd1bcdad1b1437a511609e
I20260929 18:39:46.776114 20038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:46.782400 20038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412: Bootstrap starting.
Sep 29 18:39:46 dist-test-slave-q5qx krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:46.785015 20038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:46.785745 20038 log.cc:824] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:46.788074 20038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412: No bootstrap required, opened a new log
I20260929 18:39:46.789395 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:46.776418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.0.198.62:44547)
0929 18:39:46.776816 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:46.776831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:46.776867 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:39:46.777495 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:46.777511 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:46.777854 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:46.778170 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:46.778985 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:46.779988 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:46.781175 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:46.781660 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:46.781780 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:46.784603 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:46.784608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:46.784622 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:46.784650 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:46.786393 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:46.786951 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:46.786956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:46.786961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:46.787169 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:46.787481 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:46.787483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:46.787484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:46.787894 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:46.788129 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:46.788375 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 18:39:46.788667 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260929 18:39:46.791671 20038 raft_consensus.cc:348] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44547 } }
I20260929 18:39:46.791913 20038 raft_consensus.cc:374] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:46.791949 20038 raft_consensus.cc:729] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11c31f6ecd0498c981b5c8e4a59d412, State: Initialized, Role: FOLLOWER
I20260929 18:39:46.792356 20038 consensus_queue.cc:260] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44547 } }
I20260929 18:39:46.792471 20038 raft_consensus.cc:388] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:46.792529 20038 raft_consensus.cc:482] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:46.792615 20038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:46.793576 20038 raft_consensus.cc:504] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44547 } }
I20260929 18:39:46.793895 20038 leader_election.cc:302] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d11c31f6ecd0498c981b5c8e4a59d412; no voters:
I20260929 18:39:46.794200 20038 leader_election.cc:288] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:46.794345 20043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:46.794596 20043 raft_consensus.cc:686] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [term 1 LEADER]: Becoming Leader. State: Replica: d11c31f6ecd0498c981b5c8e4a59d412, State: Running, Role: LEADER
I20260929 18:39:46.794922 20043 consensus_queue.cc:237] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44547 } }
I20260929 18:39:46.795367 20038 sys_catalog.cc:564] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:46.796506 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44547 } } }
I20260929 18:39:46.796783 20045 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d11c31f6ecd0498c981b5c8e4a59d412. Latest consensus state: current_term: 1 leader_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11c31f6ecd0498c981b5c8e4a59d412" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44547 } } }
I20260929 18:39:46.796918 20045 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:46.796986 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:46.797720 20057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:46.800314 20057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:46.805501 20057 catalog_manager.cc:1347] Generated new cluster ID: 72c407c9dcef4a0e9122b605199968d8
I20260929 18:39:46.805575 20057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:46.818608 20057 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:46.819551 20057 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:46.824388 20057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412: Generated new TSK 0
I20260929 18:39:46.825227 20057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
Entry for principal kudu/127.0.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab.
I20260929 18:39:46.864847 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44547
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:46.974164 20064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:46.974432 20064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:46.974490 20064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:46.978502 20064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:46.978567 20064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:46.978657 20064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:46.984086 20064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.1.keytab
--principal=kudu/127.0.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:46.985211 20064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:46.986264 20064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:46.996263 20072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:46.996273 20069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:46.996452 20064 server_base.cc:1034] running on GCE node
W20260929 18:39:46.996279 20070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:46.996955 20064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:46.997677 20064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:46.998855 20064 hybrid_clock.cc:648] HybridClock initialized: now 1790707186998833 us; error 33 us; skew 500 ppm
Sep 29 18:39:46 dist-test-slave-q5qx krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:47.001626 20064 init.cc:377] Logged in from keytab as kudu/127.0.198.1@KRBTEST.COM (short username kudu)
I20260929 18:39:47.002583 20064 webserver.cc:460] Webserver started at http://127.0.198.1:46337/ using document root <none> and password file <none>
I20260929 18:39:47.003154 20064 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:47.003223 20064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:47.003437 20064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:47.005550 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "bdf4a217dcff418d965a968c626ec6fb"
format_stamp: "Formatted at 2026-09-29 18:39:46 on dist-test-slave-q5qx"
server_key: "3350f84488390718a1ca1b73b27cfb52"
server_key_iv: "fbcd71e1bcbdc15c7cece26d5a9ba034"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.006147 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "bdf4a217dcff418d965a968c626ec6fb"
format_stamp: "Formatted at 2026-09-29 18:39:46 on dist-test-slave-q5qx"
server_key: "3350f84488390718a1ca1b73b27cfb52"
server_key_iv: "fbcd71e1bcbdc15c7cece26d5a9ba034"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.009995 20064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:39:47.012478 20079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.013774 20064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:47.013885 20064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "bdf4a217dcff418d965a968c626ec6fb"
format_stamp: "Formatted at 2026-09-29 18:39:46 on dist-test-slave-q5qx"
server_key: "3350f84488390718a1ca1b73b27cfb52"
server_key_iv: "fbcd71e1bcbdc15c7cece26d5a9ba034"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.014029 20064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:47.050314 20064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:47.053586 20064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:47.053762 20064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:47.054385 20064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:47.055203 20064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:47.055248 20064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.055287 20064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:47.055313 20064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.063853 20064 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:37225
I20260929 18:39:47.063875 20192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:37225 every 8 connection(s)
I20260929 18:39:47.064733 20064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:47.071801 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 20064
I20260929 18:39:47.071933 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:47.072211 792 external_mini_cluster.cc:1428] Setting key 197ad26ea2132d328be031599856d178
Sep 29 18:39:47 dist-test-slave-q5qx krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.1@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:47.079035 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.066555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:57259 (local address 127.0.198.62:44547)
0929 18:39:47.066691 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.066694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.067518 (+ 824us) server_negotiation.cc:407] Connection header received
0929 18:39:47.068805 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.068808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.068856 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.068931 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:47.070193 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.070754 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.071919 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.072206 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.072294 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.075100 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:47.075104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:47.075107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:47.075141 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:47.076574 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.077232 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.077235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.077237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.077323 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.077768 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.077771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.077773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.077964 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:47.078095 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.078787 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.078908 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:39:47.080231 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.066914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44547 (local address 127.0.198.1:57259)
0929 18:39:47.067360 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.067404 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.068580 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.069086 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.069092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.069497 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:47.070025 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.070047 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.070858 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.070860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.071783 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.071792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.072360 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.072364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.072528 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.073131 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:47.073148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:47.074902 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:47.076686 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:47.076693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:47.076711 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:47.077067 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:47.077426 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:47.077430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:47.077432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:47.077641 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:47.078112 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:47.078120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:47.078482 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.078730 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.079108 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.198.2@KRBTEST.COM; defaulting to no policy
I20260929 18:39:47.081570 20193 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44547
I20260929 18:39:47.081830 20193 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:47.082393 20193 heartbeater.cc:511] Master 127.0.198.62:44547 requested a full tablet report, sending...
I20260929 18:39:47.083957 20002 ts_manager.cc:194] Registered new tserver with Master: bdf4a217dcff418d965a968c626ec6fb (127.0.198.1:37225)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.2@KRBTEST.COM" created.
I20260929 18:39:47.085109 20002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.1@KRBTEST.COM'} at 127.0.198.1:57259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
Entry for principal kudu/127.0.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab.
I20260929 18:39:47.101480 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44547
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:47.211272 20198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:47.211519 20198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:47.211560 20198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:47.215517 20198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:47.215576 20198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:47.215634 20198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:47.221167 20198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.2.keytab
--principal=kudu/127.0.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:47.222255 20198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:47.223273 20198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:47.233157 20206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:47.233208 20198 server_base.cc:1034] running on GCE node
W20260929 18:39:47.233237 20203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:47.233167 20204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:47.233750 20198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:47.234408 20198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:47.235582 20198 hybrid_clock.cc:648] HybridClock initialized: now 1790707187235560 us; error 33 us; skew 500 ppm
Sep 29 18:39:47 dist-test-slave-q5qx krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:47.238430 20198 init.cc:377] Logged in from keytab as kudu/127.0.198.2@KRBTEST.COM (short username kudu)
I20260929 18:39:47.239323 20198 webserver.cc:460] Webserver started at http://127.0.198.2:40233/ using document root <none> and password file <none>
I20260929 18:39:47.239866 20198 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:47.239931 20198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:47.240130 20198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:47.242311 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "688c992544f2427f8b1752fad38dd3b4"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "11cd30d1a4769f2380a1d834a04a3326"
server_key_iv: "09d8b20dae26fc5250bdb5da6da3f725"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.242861 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "688c992544f2427f8b1752fad38dd3b4"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "11cd30d1a4769f2380a1d834a04a3326"
server_key_iv: "09d8b20dae26fc5250bdb5da6da3f725"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.246742 20198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:39:47.249305 20213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.250624 20198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:47.250743 20198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "688c992544f2427f8b1752fad38dd3b4"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "11cd30d1a4769f2380a1d834a04a3326"
server_key_iv: "09d8b20dae26fc5250bdb5da6da3f725"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.250852 20198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:47.261162 20198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:47.264318 20198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:47.264494 20198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:47.265084 20198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:47.265890 20198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:47.265937 20198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.266009 20198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:47.266040 20198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.274583 20198 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:41491
I20260929 18:39:47.274605 20326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:41491 every 8 connection(s)
I20260929 18:39:47.275444 20198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:47.277602 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 20198
I20260929 18:39:47.277729 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:39:47.278057 792 external_mini_cluster.cc:1428] Setting key 3be71afb8e5cb509aa8bf21e8a60190c
Sep 29 18:39:47 dist-test-slave-q5qx krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.2@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.3@KRBTEST.COM" created.
I20260929 18:39:47.290171 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.277293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:50689 (local address 127.0.198.62:44547)
0929 18:39:47.277469 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.277474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.278304 (+ 830us) server_negotiation.cc:407] Connection header received
0929 18:39:47.279733 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.279736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.279788 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.279876 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:47.281109 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.281895 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.283150 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.283500 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.283611 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.286312 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:47.286317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:47.286320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:47.286350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:47.287891 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.288432 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.288435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.288436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.288510 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.288916 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.288920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.288921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.289164 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:47.289288 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.289848 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.290004 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:39:47.291018 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.277627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44547 (local address 127.0.198.2:50689)
0929 18:39:47.278146 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.278199 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.279532 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.280130 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.280137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.280485 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:47.280973 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.280987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.282103 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.282106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.283007 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.283018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.283665 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.283668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.283813 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.284414 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:47.284430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:47.286142 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:47.288009 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:47.288014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:47.288035 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:47.288324 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:47.288616 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:47.288619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:47.288620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:47.288806 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:47.289300 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:47.289307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:47.289591 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.289768 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.290131 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260929 18:39:47.292218 20327 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44547
I20260929 18:39:47.292476 20327 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:47.293074 20327 heartbeater.cc:511] Master 127.0.198.62:44547 requested a full tablet report, sending...
I20260929 18:39:47.294250 20002 ts_manager.cc:194] Registered new tserver with Master: 688c992544f2427f8b1752fad38dd3b4 (127.0.198.2:41491)
I20260929 18:39:47.294847 20002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.2@KRBTEST.COM'} at 127.0.198.2:50689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
Entry for principal kudu/127.0.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab.
I20260929 18:39:47.306080 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44547
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:47.417009 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:47.417276 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:47.417318 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:47.421484 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:47.421545 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:47.421605 20332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:47.426952 20332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/krb5kdc/kudu_127.0.198.3.keytab
--principal=kudu/127.0.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:47.428181 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:47.429344 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:47.438921 20337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:47.438905 20338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:47.438925 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:47.439101 20332 server_base.cc:1034] running on GCE node
I20260929 18:39:47.439611 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:47.440265 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:47.441426 20332 hybrid_clock.cc:648] HybridClock initialized: now 1790707187441403 us; error 38 us; skew 500 ppm
Sep 29 18:39:47 dist-test-slave-q5qx krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:47.444211 20332 init.cc:377] Logged in from keytab as kudu/127.0.198.3@KRBTEST.COM (short username kudu)
I20260929 18:39:47.445082 20332 webserver.cc:460] Webserver started at http://127.0.198.3:32835/ using document root <none> and password file <none>
I20260929 18:39:47.445616 20332 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:47.445680 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:47.445843 20332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:47.447880 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "c49c47380cca4bacaba801882ec56c58"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "ef1d12737c6d35a7ad69112ef1c09c4a"
server_key_iv: "39034df1bb1843be46202c60d0c60f9f"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.448397 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "c49c47380cca4bacaba801882ec56c58"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "ef1d12737c6d35a7ad69112ef1c09c4a"
server_key_iv: "39034df1bb1843be46202c60d0c60f9f"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.452008 20332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 18:39:47.454502 20347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.455796 20332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:47.455890 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "c49c47380cca4bacaba801882ec56c58"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "ef1d12737c6d35a7ad69112ef1c09c4a"
server_key_iv: "39034df1bb1843be46202c60d0c60f9f"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.455997 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:47.469292 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:47.472366 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:47.472530 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:47.473099 20332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:47.473909 20332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:47.473954 20332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.474026 20332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:47.474057 20332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.482601 20332 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:38391
I20260929 18:39:47.482653 20460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:38391 every 8 connection(s)
I20260929 18:39:47.483515 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:47.492020 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 20332
I20260929 18:39:47.492167 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:47.492475 792 external_mini_cluster.cc:1428] Setting key c537385956471f8d87433b04dbeab660
Sep 29 18:39:47 dist-test-slave-q5qx krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.3@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:39:47.497432 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.485373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:51875 (local address 127.0.198.62:44547)
0929 18:39:47.485513 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.485517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.486412 (+ 895us) server_negotiation.cc:407] Connection header received
0929 18:39:47.487715 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.487718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.487767 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.487842 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:47.489087 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.489646 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.490844 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.491079 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.491182 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.493933 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:47.493935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:47.493938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:47.493963 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:47.495391 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.495874 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.495877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.495879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.495956 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.496328 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.496331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.496333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.496517 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:47.496600 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.497111 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.497267 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:39:47.498258 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.485739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44547 (local address 127.0.198.3:51875)
0929 18:39:47.486259 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.486303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.487485 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.488000 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.488007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.488416 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:47.488919 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.488937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.489745 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.489747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.490712 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.490722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.491246 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.491249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.491397 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.491957 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:47.491984 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:47.493769 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:47.495489 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:47.495496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:47.495512 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:47.495783 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:47.496036 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:47.496039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:47.496041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:47.496233 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:47.496623 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:47.496630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:47.496875 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.497046 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.497395 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260929 18:39:47.499269 20461 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44547
I20260929 18:39:47.499508 20461 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:47.500085 20461 heartbeater.cc:511] Master 127.0.198.62:44547 requested a full tablet report, sending...
I20260929 18:39:47.501106 20002 ts_manager.cc:194] Registered new tserver with Master: c49c47380cca4bacaba801882ec56c58 (127.0.198.3:38391)
I20260929 18:39:47.501649 20002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.198.3@KRBTEST.COM'} at 127.0.198.3:51875
I20260929 18:39:47.506500 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:47.515594 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.508431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.0.198.62:44547)
0929 18:39:47.508586 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.508589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.508661 (+ 72us) server_negotiation.cc:407] Connection header received
0929 18:39:47.508825 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.508827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.508873 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.508941 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:47.509598 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.510250 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.511100 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.511322 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.511401 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.512776 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:47.512779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:47.512780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:47.512804 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:47.514083 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.514440 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.514443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.514445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.514534 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:47.514775 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:47.514779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:47.514780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:47.515021 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:47.515143 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.515311 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.515412 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:39:47.518571 20002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:47.521659 20002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:47.536417 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.530358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37225 (local address 127.0.0.1:50648)
0929 18:39:47.531300 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.531340 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.531468 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.532171 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.532178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.532200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:47.532353 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.532360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.533676 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.533680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.534934 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.534943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.535856 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.535859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.535978 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.536019 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.536125 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.536252 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":56,"spinlock_wait_cycles":8064,"thread_start_us":89,"threads_started":1}
I20260929 18:39:47.536445 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.530604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.0.198.1:37225)
0929 18:39:47.530955 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.530972 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.531616 (+ 644us) server_negotiation.cc:407] Connection header received
0929 18:39:47.531792 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.531797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.531962 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.532120 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:47.532459 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.533545 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.535059 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.535734 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.535837 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.535981 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.536166 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.536275 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":82,"threads_started":1}
I20260929 18:39:47.536899 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.530287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:41491 (local address 127.0.0.1:41616)
0929 18:39:47.531288 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.531338 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.531468 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.532444 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.532448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.532465 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:47.532570 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.532576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.534088 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.534091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.535325 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.535335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.536511 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.536514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.536620 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.536642 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.536704 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.536766 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":16,"spinlock_wait_cycles":1920,"thread_start_us":96,"threads_started":1}
I20260929 18:39:47.537128 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.531309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:38391 (local address 127.0.0.1:37096)
0929 18:39:47.532708 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.532719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.532797 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.533169 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.533172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.533187 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:47.533288 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.533295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.534536 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.534540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.535595 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.535602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.536744 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.536748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.536846 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.536868 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.536930 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.537002 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260929 18:39:47.537532 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.531694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.0.198.3:38391)
0929 18:39:47.532023 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.532035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.532798 (+ 763us) server_negotiation.cc:407] Connection header received
0929 18:39:47.532897 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.532903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.533051 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.533171 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:47.533376 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.534421 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.535699 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.536443 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.536539 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.536687 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.537286 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.537382 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":176,"threads_started":1}
I20260929 18:39:47.537521 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.530447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.0.198.2:41491)
0929 18:39:47.532048 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.532063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.532088 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:39:47.532164 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.532169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.532319 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.532455 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:47.532695 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.533910 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.535456 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.536210 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.536309 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.536443 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.537178 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.537365 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":110,"threads_started":1}
I20260929 18:39:47.539499 20127 tablet_service.cc:1467] Processing CreateTablet for tablet 29b10db0cd744090a22b22691f0ded3c (DEFAULT_TABLE table=test-table [id=dccee8aed914411f8fa5d3f1e87b131b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:47.540469 20127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b10db0cd744090a22b22691f0ded3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:47.540215 20261 tablet_service.cc:1467] Processing CreateTablet for tablet 29b10db0cd744090a22b22691f0ded3c (DEFAULT_TABLE table=test-table [id=dccee8aed914411f8fa5d3f1e87b131b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:47.540215 20395 tablet_service.cc:1467] Processing CreateTablet for tablet 29b10db0cd744090a22b22691f0ded3c (DEFAULT_TABLE table=test-table [id=dccee8aed914411f8fa5d3f1e87b131b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:47.541235 20395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b10db0cd744090a22b22691f0ded3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:47.541257 20261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b10db0cd744090a22b22691f0ded3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:47.547596 20478 tablet_bootstrap.cc:492] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb: Bootstrap starting.
I20260929 18:39:47.548640 20479 tablet_bootstrap.cc:492] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Bootstrap starting.
I20260929 18:39:47.549551 20478 tablet_bootstrap.cc:654] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:47.550300 20480 tablet_bootstrap.cc:492] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Bootstrap starting.
I20260929 18:39:47.550388 20479 tablet_bootstrap.cc:654] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:47.550632 20478 log.cc:824] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:47.551515 20479 log.cc:824] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:47.552596 20478 tablet_bootstrap.cc:492] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb: No bootstrap required, opened a new log
I20260929 18:39:47.552623 20480 tablet_bootstrap.cc:654] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:47.552780 20478 ts_tablet_manager.cc:1397] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:47.553383 20479 tablet_bootstrap.cc:492] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: No bootstrap required, opened a new log
I20260929 18:39:47.553359 20480 log.cc:824] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:47.553550 20479 ts_tablet_manager.cc:1397] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:39:47.555289 20480 tablet_bootstrap.cc:492] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: No bootstrap required, opened a new log
I20260929 18:39:47.555461 20480 ts_tablet_manager.cc:1397] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:39:47.555900 20478 raft_consensus.cc:348] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.556113 20478 raft_consensus.cc:374] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:47.556164 20478 raft_consensus.cc:729] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf4a217dcff418d965a968c626ec6fb, State: Initialized, Role: FOLLOWER
I20260929 18:39:47.556557 20478 consensus_queue.cc:260] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.557361 20478 ts_tablet_manager.cc:1428] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:39:47.557211 20479 raft_consensus.cc:348] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.557457 20479 raft_consensus.cc:374] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:47.557518 20479 raft_consensus.cc:729] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49c47380cca4bacaba801882ec56c58, State: Initialized, Role: FOLLOWER
I20260929 18:39:47.557731 20193 heartbeater.cc:503] Master 127.0.198.62:44547 was elected leader, sending a full tablet report...
I20260929 18:39:47.557940 20479 consensus_queue.cc:260] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.558774 20461 heartbeater.cc:503] Master 127.0.198.62:44547 was elected leader, sending a full tablet report...
I20260929 18:39:47.558883 20479 ts_tablet_manager.cc:1428] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:39:47.559890 20480 raft_consensus.cc:348] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.560163 20480 raft_consensus.cc:374] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:47.560220 20480 raft_consensus.cc:729] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688c992544f2427f8b1752fad38dd3b4, State: Initialized, Role: FOLLOWER
I20260929 18:39:47.560624 20480 consensus_queue.cc:260] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.561224 20327 heartbeater.cc:503] Master 127.0.198.62:44547 was elected leader, sending a full tablet report...
I20260929 18:39:47.561491 20480 ts_tablet_manager.cc:1428] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 18:39:47.565788 20194 tablet.cc:2369] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:47.680260 20486 raft_consensus.cc:482] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:47.680490 20486 raft_consensus.cc:504] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.681700 20486 leader_election.cc:288] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdf4a217dcff418d965a968c626ec6fb (127.0.198.1:37225), c49c47380cca4bacaba801882ec56c58 (127.0.198.3:38391)
I20260929 18:39:47.681702 20485 raft_consensus.cc:482] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:47.681869 20485 raft_consensus.cc:504] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.683647 20485 leader_election.cc:288] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 688c992544f2427f8b1752fad38dd3b4 (127.0.198.2:41491), bdf4a217dcff418d965a968c626ec6fb (127.0.198.1:37225)
I20260929 18:39:47.686749 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.682180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:52241 (local address 127.0.198.3:38391)
0929 18:39:47.682304 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.682308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.682323 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:39:47.682369 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.682372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.682416 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.682503 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:47.682866 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.683566 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.685431 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.686124 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.686212 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.686270 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.686539 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.686613 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":47}
I20260929 18:39:47.686749 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.681885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:38391 (local address 127.0.198.2:52241)
0929 18:39:47.682113 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.682128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.682218 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.682531 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.682534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.682554 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:47.682699 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.682706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.684027 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.684030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.685291 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.685301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.686305 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.686309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.686416 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.686439 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.686516 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.686597 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":2}
I20260929 18:39:47.687218 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.683220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:41491 (local address 127.0.198.3:55699)
0929 18:39:47.683436 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.683449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.683509 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.683960 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.683963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.683979 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:47.684070 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.684089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.684996 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.684999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.686059 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.686068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.686843 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.686846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.686936 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.686955 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.687027 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.687101 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":28}
I20260929 18:39:47.687439 20415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "688c992544f2427f8b1752fad38dd3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c49c47380cca4bacaba801882ec56c58" is_pre_election: true
I20260929 18:39:47.687619 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.683587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:55699 (local address 127.0.198.2:41491)
0929 18:39:47.683717 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.683721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.683732 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:39:47.683767 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.683769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.683807 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.683873 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:47.684192 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.684882 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.686197 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.686635 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.686731 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.686783 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.687455 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.687518 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:39:47.687657 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.682087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:40315 (local address 127.0.198.1:37225)
0929 18:39:47.682297 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.682302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.682805 (+ 503us) server_negotiation.cc:407] Connection header received
0929 18:39:47.682923 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.682926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.682964 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.683038 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:47.683415 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.684086 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.685449 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.686060 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.686165 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.686229 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.687414 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.687532 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":139}
I20260929 18:39:47.687680 20415 raft_consensus.cc:2454] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688c992544f2427f8b1752fad38dd3b4 in term 0.
I20260929 18:39:47.688133 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.681885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37225 (local address 127.0.198.2:40315)
0929 18:39:47.682707 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.682720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.682804 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.683167 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.683170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.683187 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:47.683295 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.683301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.684200 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.684203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.685335 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.685342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.687090 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.687093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.687183 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.687202 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.687479 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.688006 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":425,"thread_start_us":113,"threads_started":1}
I20260929 18:39:47.688086 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "c49c47380cca4bacaba801882ec56c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688c992544f2427f8b1752fad38dd3b4" is_pre_election: true
I20260929 18:39:47.688140 20214 leader_election.cc:302] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688c992544f2427f8b1752fad38dd3b4, c49c47380cca4bacaba801882ec56c58; no voters:
I20260929 18:39:47.688313 20281 raft_consensus.cc:2454] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c49c47380cca4bacaba801882ec56c58 in term 0.
I20260929 18:39:47.688457 20486 raft_consensus.cc:2781] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:47.688766 20350 leader_election.cc:302] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688c992544f2427f8b1752fad38dd3b4, c49c47380cca4bacaba801882ec56c58; no voters:
I20260929 18:39:47.688815 20486 raft_consensus.cc:482] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:47.688879 20486 raft_consensus.cc:3037] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:47.688942 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.683637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:34409 (local address 127.0.198.1:37225)
0929 18:39:47.683898 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.683902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.684062 (+ 160us) server_negotiation.cc:407] Connection header received
0929 18:39:47.684107 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.684109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.684152 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.684220 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:47.684535 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.685245 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.687370 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.688124 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.688321 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.688374 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.688754 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.688823 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
I20260929 18:39:47.689136 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.683254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37225 (local address 127.0.198.3:34409)
0929 18:39:47.683905 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.683918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:47.683992 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:47.684304 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:47.684307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:47.684321 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:47.684416 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.684421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.685360 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.685364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.687165 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:47.687174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.688409 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.688413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:47.688502 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.688653 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 18:39:47.688724 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:39:47.688789 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":400,"thread_start_us":99,"threads_started":1}
I20260929 18:39:47.689257 20147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "688c992544f2427f8b1752fad38dd3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf4a217dcff418d965a968c626ec6fb" is_pre_election: true
I20260929 18:39:47.689524 20485 raft_consensus.cc:2781] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:47.689535 20147 raft_consensus.cc:2454] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688c992544f2427f8b1752fad38dd3b4 in term 0.
I20260929 18:39:47.689602 20485 raft_consensus.cc:482] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:47.689649 20485 raft_consensus.cc:3037] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:47.689512 20146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "c49c47380cca4bacaba801882ec56c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf4a217dcff418d965a968c626ec6fb" is_pre_election: true
I20260929 18:39:47.689815 20146 raft_consensus.cc:2454] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c49c47380cca4bacaba801882ec56c58 in term 0.
I20260929 18:39:47.690563 20486 raft_consensus.cc:504] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.690689 20485 raft_consensus.cc:504] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.690896 20486 leader_election.cc:288] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [CANDIDATE]: Term 1 election: Requested vote from peers bdf4a217dcff418d965a968c626ec6fb (127.0.198.1:37225), c49c47380cca4bacaba801882ec56c58 (127.0.198.3:38391)
I20260929 18:39:47.691233 20415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "688c992544f2427f8b1752fad38dd3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c49c47380cca4bacaba801882ec56c58"
I20260929 18:39:47.691323 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "c49c47380cca4bacaba801882ec56c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688c992544f2427f8b1752fad38dd3b4"
I20260929 18:39:47.691329 20147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "c49c47380cca4bacaba801882ec56c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf4a217dcff418d965a968c626ec6fb"
I20260929 18:39:47.691326 20146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b10db0cd744090a22b22691f0ded3c" candidate_uuid: "688c992544f2427f8b1752fad38dd3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf4a217dcff418d965a968c626ec6fb"
I20260929 18:39:47.691425 20415 raft_consensus.cc:2379] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 688c992544f2427f8b1752fad38dd3b4 in current term 1: Already voted for candidate c49c47380cca4bacaba801882ec56c58 in this term.
I20260929 18:39:47.691450 20281 raft_consensus.cc:2379] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c49c47380cca4bacaba801882ec56c58 in current term 1: Already voted for candidate 688c992544f2427f8b1752fad38dd3b4 in this term.
I20260929 18:39:47.691460 20147 raft_consensus.cc:3037] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:47.691249 20485 leader_election.cc:288] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [CANDIDATE]: Term 1 election: Requested vote from peers 688c992544f2427f8b1752fad38dd3b4 (127.0.198.2:41491), bdf4a217dcff418d965a968c626ec6fb (127.0.198.1:37225)
I20260929 18:39:47.691973 20214 leader_election.cc:302] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 688c992544f2427f8b1752fad38dd3b4; no voters: bdf4a217dcff418d965a968c626ec6fb, c49c47380cca4bacaba801882ec56c58
I20260929 18:39:47.692122 20486 raft_consensus.cc:2726] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 18:39:47.692488 20147 raft_consensus.cc:2454] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c49c47380cca4bacaba801882ec56c58 in term 1.
I20260929 18:39:47.692791 20351 leader_election.cc:302] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdf4a217dcff418d965a968c626ec6fb, c49c47380cca4bacaba801882ec56c58; no voters: 688c992544f2427f8b1752fad38dd3b4
I20260929 18:39:47.692975 20485 raft_consensus.cc:2781] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:47.693194 20485 raft_consensus.cc:686] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 1 LEADER]: Becoming Leader. State: Replica: c49c47380cca4bacaba801882ec56c58, State: Running, Role: LEADER
I20260929 18:39:47.693523 20485 consensus_queue.cc:237] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } }
I20260929 18:39:47.696863 20002 catalog_manager.cc:5495] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to c49c47380cca4bacaba801882ec56c58 (127.0.198.3). New cstate: current_term: 1 leader_uuid: "c49c47380cca4bacaba801882ec56c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c49c47380cca4bacaba801882ec56c58" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260929 18:39:47.725558 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.722123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.198.3:38391)
0929 18:39:47.722269 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.722274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.722287 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:39:47.722367 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.722370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.722424 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.722512 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:47.722773 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.723304 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.724154 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.724374 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.724459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.724756 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:47.725186 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:47.725296 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.725365 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.725420 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:39:47.731709 20281 raft_consensus.cc:1261] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 1 FOLLOWER]: Refusing update from remote peer c49c47380cca4bacaba801882ec56c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:47.731709 20147 raft_consensus.cc:1261] T 29b10db0cd744090a22b22691f0ded3c P bdf4a217dcff418d965a968c626ec6fb [term 1 FOLLOWER]: Refusing update from remote peer c49c47380cca4bacaba801882ec56c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:47.732304 20490 consensus_queue.cc:1035] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688c992544f2427f8b1752fad38dd3b4" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:47.732539 20485 consensus_queue.cc:1035] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf4a217dcff418d965a968c626ec6fb" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:39:47.734366 20462 tablet.cc:2369] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:47.738932 20493 mvcc.cc:205] Tried to move back new op lower bound from 7334736640942280704 to 7334736640795443200. Current Snapshot: MvccSnapshot[applied={T|T < 7334736640942280704}]
I20260929 18:39:47.747709 20002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:47.747869 20002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:47.750190 20002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d11c31f6ecd0498c981b5c8e4a59d412: Sending DeleteTablet for 3 replicas of tablet 29b10db0cd744090a22b22691f0ded3c
I20260929 18:39:47.751037 20261 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b10db0cd744090a22b22691f0ded3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:47 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:41616
I20260929 18:39:47.751025 20127 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b10db0cd744090a22b22691f0ded3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:47 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:50648
I20260929 18:39:47.751502 20503 tablet_replica.cc:330] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: stopping tablet replica
I20260929 18:39:47.751659 20395 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b10db0cd744090a22b22691f0ded3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:47 UTC) from {username='kudu', principal='kudu/127.0.198.62@KRBTEST.COM'} at 127.0.0.1:37096
I20260929 18:39:47.751957 20503 raft_consensus.cc:2229] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:47.752111 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 20064
I20260929 18:39:47.752326 20503 raft_consensus.cc:2258] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:47.752691 20505 tablet_replica.cc:330] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: stopping tablet replica
I20260929 18:39:47.752956 20505 raft_consensus.cc:2229] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:39:47.753253 20505 raft_consensus.cc:2258] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:47.754065 20503 ts_tablet_manager.cc:1905] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:47.754784 20505 ts_tablet_manager.cc:1905] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:47.756767 20503 ts_tablet_manager.cc:1918] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:47.756850 20503 log.cc:1196] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/29b10db0cd744090a22b22691f0ded3c
I20260929 18:39:47.757392 20505 ts_tablet_manager.cc:1918] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:47.757443 20503 ts_tablet_manager.cc:1939] T 29b10db0cd744090a22b22691f0ded3c P 688c992544f2427f8b1752fad38dd3b4: Deleting consensus metadata
I20260929 18:39:47.757467 20505 log.cc:1196] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/29b10db0cd744090a22b22691f0ded3c
I20260929 18:39:47.757756 20505 ts_tablet_manager.cc:1939] T 29b10db0cd744090a22b22691f0ded3c P c49c47380cca4bacaba801882ec56c58: Deleting consensus metadata
I20260929 18:39:47.758692 19988 catalog_manager.cc:4841] TS 688c992544f2427f8b1752fad38dd3b4 (127.0.198.2:41491): tablet 29b10db0cd744090a22b22691f0ded3c (table test-table [id=dccee8aed914411f8fa5d3f1e87b131b]) successfully deleted
I20260929 18:39:47.758994 19986 catalog_manager.cc:4841] TS c49c47380cca4bacaba801882ec56c58 (127.0.198.3:38391): tablet 29b10db0cd744090a22b22691f0ded3c (table test-table [id=dccee8aed914411f8fa5d3f1e87b131b]) successfully deleted
W20260929 18:39:47.761399 19989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:37225 (error 108)
I20260929 18:39:47.762356 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.762093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37225 (local address 127.0.0.1:50662)
0929 18:39:47.762208 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:47.762271 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 18:39:47.762476 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 20198
I20260929 18:39:47.767239 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 20332
W20260929 18:39:47.767319 19989 catalog_manager.cc:4584] TS bdf4a217dcff418d965a968c626ec6fb (127.0.198.1:37225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29b10db0cd744090a22b22691f0ded3c: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37225: connect: Connection refused (error 111)
I20260929 18:39:47.772096 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 19970
2026-09-29T18:39:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2275 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T18:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:47Z Disabled control of system clock
I20260929 18:39:47.799583 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44141
--webserver_interface=127.0.198.62
--webserver_port=0
--builtin_ntp_servers=127.0.198.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:44141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:39:47.905992 20510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:47.906270 20510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:47.906306 20510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:47.910252 20510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:47.910305 20510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:47.910319 20510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:47.910336 20510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:47.910348 20510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:47.914994 20510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:44141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:44141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:47.916010 20510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:47.917359 20510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:47.925815 20515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:47.925874 20510 server_base.cc:1034] running on GCE node
W20260929 18:39:47.925835 20516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:47.925832 20518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:47.926695 20510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:47.927928 20510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:47.929199 20510 hybrid_clock.cc:648] HybridClock initialized: now 1790707187929183 us; error 34 us; skew 500 ppm
I20260929 18:39:47.931241 20510 webserver.cc:460] Webserver started at http://127.0.198.62:42073/ using document root <none> and password file <none>
I20260929 18:39:47.931821 20510 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:47.931890 20510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:47.932193 20510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:47.934315 20510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "b240bf8530bd4979a7fecf9bf86aef38"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "6dbea28f5b947a17e44c939ec5b3af9b"
server_key_iv: "d2ecad43ac5057fc8e9d4f4aeb9aaa3e"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.934863 20510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "b240bf8530bd4979a7fecf9bf86aef38"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "6dbea28f5b947a17e44c939ec5b3af9b"
server_key_iv: "d2ecad43ac5057fc8e9d4f4aeb9aaa3e"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.938621 20510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:39:47.940994 20524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:47.942404 20510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:47.942499 20510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "b240bf8530bd4979a7fecf9bf86aef38"
format_stamp: "Formatted at 2026-09-29 18:39:47 on dist-test-slave-q5qx"
server_key: "6dbea28f5b947a17e44c939ec5b3af9b"
server_key_iv: "d2ecad43ac5057fc8e9d4f4aeb9aaa3e"
server_key_version: "encryptionkey@0"
I20260929 18:39:47.942593 20510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:47.956102 20510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:47.956753 20510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:47.956909 20510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:47.963316 20510 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:44141
I20260929 18:39:47.963330 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:44141 every 8 connection(s)
I20260929 18:39:47.964221 20510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:47.964643 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 20510
I20260929 18:39:47.964757 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:47.965023 792 external_mini_cluster.cc:1428] Setting key 479488a571be503dce66b9b4ef9985b1
I20260929 18:39:47.967465 20577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:47.973517 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38: Bootstrap starting.
I20260929 18:39:47.974274 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:47.966478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.0.198.62:44141)
0929 18:39:47.967082 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:47.967100 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:47.967137 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:39:47.967794 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:47.967812 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:47.968266 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:47.968564 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:47.968896 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.969895 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.971092 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:47.971558 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:47.971679 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:47.972558 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:47.972563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:47.972574 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:47.972962 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:47.973017 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:47.973037 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:47.973111 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:47.973256 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:39:47.973511 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":124,"threads_started":1}
I20260929 18:39:47.975829 20577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:47.976588 20577 log.cc:824] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:47.978494 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38: No bootstrap required, opened a new log
I20260929 18:39:47.981773 20577 raft_consensus.cc:348] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b240bf8530bd4979a7fecf9bf86aef38" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44141 } }
I20260929 18:39:47.982010 20577 raft_consensus.cc:374] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:47.982069 20577 raft_consensus.cc:729] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b240bf8530bd4979a7fecf9bf86aef38, State: Initialized, Role: FOLLOWER
I20260929 18:39:47.982527 20577 consensus_queue.cc:260] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b240bf8530bd4979a7fecf9bf86aef38" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44141 } }
I20260929 18:39:47.982658 20577 raft_consensus.cc:388] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:47.982720 20577 raft_consensus.cc:482] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:47.982816 20577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:47.983666 20577 raft_consensus.cc:504] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b240bf8530bd4979a7fecf9bf86aef38" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44141 } }
I20260929 18:39:47.983979 20577 leader_election.cc:302] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b240bf8530bd4979a7fecf9bf86aef38; no voters:
I20260929 18:39:47.984256 20577 leader_election.cc:288] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:47.984393 20582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:47.984596 20582 raft_consensus.cc:686] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [term 1 LEADER]: Becoming Leader. State: Replica: b240bf8530bd4979a7fecf9bf86aef38, State: Running, Role: LEADER
I20260929 18:39:47.984874 20582 consensus_queue.cc:237] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b240bf8530bd4979a7fecf9bf86aef38" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44141 } }
I20260929 18:39:47.985378 20577 sys_catalog.cc:564] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:47.986419 20583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b240bf8530bd4979a7fecf9bf86aef38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b240bf8530bd4979a7fecf9bf86aef38" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44141 } } }
I20260929 18:39:47.986423 20584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b240bf8530bd4979a7fecf9bf86aef38. Latest consensus state: current_term: 1 leader_uuid: "b240bf8530bd4979a7fecf9bf86aef38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b240bf8530bd4979a7fecf9bf86aef38" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 44141 } } }
I20260929 18:39:47.986550 20584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:47.986549 20583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:47.987113 20591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:47.990239 20591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:47.995880 20591 catalog_manager.cc:1347] Generated new cluster ID: cc943d641bb7468f8e0f66d7c906a1be
I20260929 18:39:47.995959 20591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:48.020339 20591 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:48.021301 20591 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:48.035828 20591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38: Generated new TSK 0
I20260929 18:39:48.036736 20591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:48.049274 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.1:0
--local_ip_for_outbound_sockets=127.0.198.1
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44141
--builtin_ntp_servers=127.0.198.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:39:48.159874 20601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:48.160136 20601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:48.160189 20601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:48.164198 20601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:48.164253 20601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:48.164326 20601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.1
I20260929 18:39:48.169020 20601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.198.1
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.1
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:48.170104 20601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:48.171454 20601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:48.180986 20606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:48.181036 20609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:48.180995 20607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:48.181169 20601 server_base.cc:1034] running on GCE node
I20260929 18:39:48.181699 20601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:48.182327 20601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:48.183478 20601 hybrid_clock.cc:648] HybridClock initialized: now 1790707188183448 us; error 42 us; skew 500 ppm
I20260929 18:39:48.185372 20601 webserver.cc:460] Webserver started at http://127.0.198.1:36753/ using document root <none> and password file <none>
I20260929 18:39:48.185920 20601 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:48.186010 20601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:48.186211 20601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:48.188247 20601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data/instance:
uuid: "555387434b2c4853a4f7914ae18a6328"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "61765cce418e3e59506d09baf99415f8"
server_key_iv: "623e5644c8e37f824bbeb0687c63374f"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.188745 20601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/wal/instance:
uuid: "555387434b2c4853a4f7914ae18a6328"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "61765cce418e3e59506d09baf99415f8"
server_key_iv: "623e5644c8e37f824bbeb0687c63374f"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.192297 20601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:39:48.194612 20615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.195930 20601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:48.196044 20601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/wal
uuid: "555387434b2c4853a4f7914ae18a6328"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "61765cce418e3e59506d09baf99415f8"
server_key_iv: "623e5644c8e37f824bbeb0687c63374f"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.196161 20601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:48.207643 20601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:48.208377 20601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:48.208608 20601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:48.209230 20601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:48.210258 20601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:48.210307 20601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.210361 20601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:48.210386 20601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.218631 20601 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.1:37293
I20260929 18:39:48.218678 20728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.1:37293 every 8 connection(s)
I20260929 18:39:48.219519 20601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:48.225564 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 20601
I20260929 18:39:48.225660 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:48.225911 792 external_mini_cluster.cc:1428] Setting key 4b5c76e46ba414737a472390d3be3fd2
I20260929 18:39:48.227990 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.2:0
--local_ip_for_outbound_sockets=127.0.198.2
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44141
--builtin_ntp_servers=127.0.198.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:39:48.229118 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.221354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.1:45105 (local address 127.0.198.62:44141)
0929 18:39:48.221504 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.221509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.222663 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 18:39:48.224066 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.224070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.224126 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.224201 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:48.225111 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.225639 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.226949 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.227217 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.227316 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.228617 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:48.228620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:48.228623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:48.228646 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:48.228675 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:48.228684 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:48.228759 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.228926 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.229005 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:39:48.230091 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.221790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44141 (local address 127.0.198.1:45105)
0929 18:39:48.222513 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.222555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.223853 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.224395 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.224403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.224493 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:48.224977 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.224990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.225775 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.225778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.226827 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.226837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.227396 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.227401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.227606 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.228296 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:48.228298 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:48.228353 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:48.228444 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:48.228773 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:48.228781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:48.228786 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.228915 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.229240 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260929 18:39:48.231382 20729 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44141
I20260929 18:39:48.231614 20729 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:48.232189 20729 heartbeater.cc:511] Master 127.0.198.62:44141 requested a full tablet report, sending...
I20260929 18:39:48.233907 20541 ts_manager.cc:194] Registered new tserver with Master: 555387434b2c4853a4f7914ae18a6328 (127.0.198.1:37293)
I20260929 18:39:48.235603 20541 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.1:45105
W20260929 18:39:48.339125 20732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:48.339399 20732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:48.339438 20732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:48.343482 20732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:48.343549 20732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:48.343621 20732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.2
I20260929 18:39:48.348551 20732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.198.2
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.2
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:48.349622 20732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:48.350993 20732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:48.360858 20738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:48.360867 20737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:48.360924 20732 server_base.cc:1034] running on GCE node
W20260929 18:39:48.360867 20740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:48.361635 20732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:48.362356 20732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:48.363538 20732 hybrid_clock.cc:648] HybridClock initialized: now 1790707188363516 us; error 36 us; skew 500 ppm
I20260929 18:39:48.365621 20732 webserver.cc:460] Webserver started at http://127.0.198.2:43155/ using document root <none> and password file <none>
I20260929 18:39:48.366246 20732 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:48.366318 20732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:48.366519 20732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:48.368577 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data/instance:
uuid: "982ff88d9dde4a518e798a28330a0ea2"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "3309f7005c64c5e937ff36de21aa2301"
server_key_iv: "5dd38f74dc4e398456925f2c2ab33d59"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.369124 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal/instance:
uuid: "982ff88d9dde4a518e798a28330a0ea2"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "3309f7005c64c5e937ff36de21aa2301"
server_key_iv: "5dd38f74dc4e398456925f2c2ab33d59"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.372864 20732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:39:48.375609 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.376914 20732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:39:48.377043 20732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal
uuid: "982ff88d9dde4a518e798a28330a0ea2"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "3309f7005c64c5e937ff36de21aa2301"
server_key_iv: "5dd38f74dc4e398456925f2c2ab33d59"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.377166 20732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:48.391790 20732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:48.392484 20732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:48.392668 20732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:48.393253 20732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:48.394210 20732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:48.394256 20732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.394312 20732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:48.394338 20732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.403499 20732 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.2:42325
I20260929 18:39:48.403524 20859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.2:42325 every 8 connection(s)
I20260929 18:39:48.404459 20732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:48.404889 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 20732
I20260929 18:39:48.405005 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal/instance
I20260929 18:39:48.405311 792 external_mini_cluster.cc:1428] Setting key 1923dd2a764eefc31dd51cf40b80092b
I20260929 18:39:48.407543 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.198.3:0
--local_ip_for_outbound_sockets=127.0.198.3
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44141
--builtin_ntp_servers=127.0.198.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:39:48.416194 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.406490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.2:47867 (local address 127.0.198.62:44141)
0929 18:39:48.406614 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.406619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.407963 (+ 1344us) server_negotiation.cc:407] Connection header received
0929 18:39:48.409633 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.409636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.409687 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.409762 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:48.410992 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.411675 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.413448 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.413770 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.413894 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.415453 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:48.415457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:48.415459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:48.415476 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:48.415500 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:48.415511 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:48.415595 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.415836 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.415940 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:39:48.417129 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.407009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44141 (local address 127.0.198.2:47867)
0929 18:39:48.407804 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.407863 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.409355 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.410007 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.410016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.410132 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:48.410817 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.410835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.411818 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.411822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.413275 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.413287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.414019 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.414026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.414360 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.415106 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:48.415108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:48.415171 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:48.415269 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:48.415638 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:48.415647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:48.415654 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.415791 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.416107 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260929 18:39:48.418476 20860 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44141
I20260929 18:39:48.418757 20860 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:48.419378 20860 heartbeater.cc:511] Master 127.0.198.62:44141 requested a full tablet report, sending...
I20260929 18:39:48.420565 20541 ts_manager.cc:194] Registered new tserver with Master: 982ff88d9dde4a518e798a28330a0ea2 (127.0.198.2:42325)
I20260929 18:39:48.421166 20541 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.2:47867
W20260929 18:39:48.521624 20863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:48.521926 20863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:48.521994 20863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:48.526201 20863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:48.526269 20863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:48.526326 20863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.198.3
I20260929 18:39:48.531361 20863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.198.3
--webserver_port=0
--tserver_master_addrs=127.0.198.62:44141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.198.3
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:48.532598 20863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:48.534162 20863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:48.544346 20869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:48.544351 20868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:48.544505 20871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:48.544796 20863 server_base.cc:1034] running on GCE node
I20260929 18:39:48.545233 20863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:48.545948 20863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:48.547236 20863 hybrid_clock.cc:648] HybridClock initialized: now 1790707188547238 us; error 56 us; skew 500 ppm
I20260929 18:39:48.549226 20863 webserver.cc:460] Webserver started at http://127.0.198.3:41223/ using document root <none> and password file <none>
I20260929 18:39:48.549834 20863 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:48.549899 20863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:48.550130 20863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:48.552295 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data/instance:
uuid: "362c5086c3df497c8955f4f4a8414055"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "bed220c15ecbef3917fb20bd90a6aaaf"
server_key_iv: "4789ab1313811d07f16038eb4e0cbd86"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.552865 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal/instance:
uuid: "362c5086c3df497c8955f4f4a8414055"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "bed220c15ecbef3917fb20bd90a6aaaf"
server_key_iv: "4789ab1313811d07f16038eb4e0cbd86"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.556846 20863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 18:39:48.559602 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.560879 20863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:39:48.560984 20863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal
uuid: "362c5086c3df497c8955f4f4a8414055"
format_stamp: "Formatted at 2026-09-29 18:39:48 on dist-test-slave-q5qx"
server_key: "bed220c15ecbef3917fb20bd90a6aaaf"
server_key_iv: "4789ab1313811d07f16038eb4e0cbd86"
server_key_version: "encryptionkey@0"
I20260929 18:39:48.561081 20863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:48.576792 20863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:48.577485 20863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:48.577674 20863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:48.578342 20863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:48.579281 20863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:48.579334 20863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.579372 20863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:48.579385 20863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:48.590555 20863 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.3:40343
I20260929 18:39:48.590582 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.3:40343 every 8 connection(s)
I20260929 18:39:48.591645 20863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:48.594662 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 20863
I20260929 18:39:48.594774 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:48.595077 792 external_mini_cluster.cc:1428] Setting key 94f80aeb74e1c5133dd10a97ba8c8085
I20260929 18:39:48.601564 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.593678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:56979 (local address 127.0.198.62:44141)
0929 18:39:48.593832 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.593837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.595153 (+ 1316us) server_negotiation.cc:407] Connection header received
0929 18:39:48.596793 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.596796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.596841 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.596911 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:48.597916 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.598469 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.599720 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.599948 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.600028 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.601175 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:48.601178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:48.601180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:48.601193 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:48.601214 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:48.601222 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:48.601278 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.601397 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.601449 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:39:48.602591 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.594168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:44141 (local address 127.0.198.3:56979)
0929 18:39:48.594972 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.595034 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.596578 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.597102 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.597111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.597237 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:48.597789 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.597804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.598585 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.598588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.599499 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.599508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.600080 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.600083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.600301 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.600923 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:39:48.600925 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:48.600967 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:39:48.601033 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:48.601288 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:48.601295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:48.601299 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.601394 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.601629 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260929 18:39:48.603969 20991 heartbeater.cc:347] Connected to a master server at 127.0.198.62:44141
I20260929 18:39:48.604223 20991 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:48.604781 20991 heartbeater.cc:511] Master 127.0.198.62:44141 requested a full tablet report, sending...
I20260929 18:39:48.605857 20541 ts_manager.cc:194] Registered new tserver with Master: 362c5086c3df497c8955f4f4a8414055 (127.0.198.3:40343)
I20260929 18:39:48.606463 20541 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.198.3:56979
I20260929 18:39:48.609339 792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:48.615173 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.611489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.0.198.62:44141)
0929 18:39:48.611655 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.611659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.611748 (+ 89us) server_negotiation.cc:407] Connection header received
0929 18:39:48.611859 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.611861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.611918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.611985 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:48.612281 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.612805 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.613605 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.613831 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.613902 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.614737 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:48.614740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:48.614743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:48.614761 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:48.614785 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:48.614793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:48.614863 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.614985 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.615033 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:39:48.618167 20541 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:48.621200 20541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:48.636426 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.629944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.198.1:37293)
0929 18:39:48.630712 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.630724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.630749 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:39:48.630832 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.630838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.631047 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.631287 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:48.631743 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.632885 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.634741 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.635611 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.635717 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.635900 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.636140 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.636259 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":131,"threads_started":1}
I20260929 18:39:48.636423 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.629812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37293 (local address 127.0.0.1:59456)
0929 18:39:48.630227 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.630275 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.630395 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.631445 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.631452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.631475 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:48.631621 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.631627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.632985 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.632988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.634617 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.634628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.635732 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.635736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.635895 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.635975 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.636093 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.636259 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":164,"thread_start_us":75,"threads_started":1}
I20260929 18:39:48.636423 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.630558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.3:40343 (local address 127.0.0.1:33802)
0929 18:39:48.630787 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.630799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.630876 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.632057 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.632059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.632076 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:48.632173 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.632180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.633360 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.633364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.634508 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.634518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.635732 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.635736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.635898 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.635975 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.636093 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.636237 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":75,"thread_start_us":92,"threads_started":1}
I20260929 18:39:48.636940 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.631088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.0.198.3:40343)
0929 18:39:48.631571 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.631579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.631597 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:39:48.631657 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.631660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.631826 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.631965 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:48.632282 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.633259 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.634624 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.635569 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.635695 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.635900 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.636650 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.636759 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":161,"threads_started":1}
I20260929 18:39:48.637959 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.631095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.0.198.2:42325)
0929 18:39:48.631392 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.631403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.631423 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:39:48.631489 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.631493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.631688 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.631828 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:48.632036 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.632832 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.635971 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.637016 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.637129 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.637333 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.637742 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.637837 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
I20260929 18:39:48.637684 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.629855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42325 (local address 127.0.0.1:46092)
0929 18:39:48.630360 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.630375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.630464 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.631809 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.631812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.631829 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:48.631937 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.631943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.632935 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.632939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.634616 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.634628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.637285 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.637288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.637380 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.637404 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.637476 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.637550 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":146,"thread_start_us":130,"threads_started":1}
I20260929 18:39:48.639801 20663 tablet_service.cc:1467] Processing CreateTablet for tablet ae45f0f92b3943bd92ec8eaa9270d0fb (DEFAULT_TABLE table=test-table [id=824497d7372b48a2a6e588a8cc6cea9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:48.640791 20663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae45f0f92b3943bd92ec8eaa9270d0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:48.640988 20794 tablet_service.cc:1467] Processing CreateTablet for tablet ae45f0f92b3943bd92ec8eaa9270d0fb (DEFAULT_TABLE table=test-table [id=824497d7372b48a2a6e588a8cc6cea9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:48.640980 20907 tablet_service.cc:1467] Processing CreateTablet for tablet ae45f0f92b3943bd92ec8eaa9270d0fb (DEFAULT_TABLE table=test-table [id=824497d7372b48a2a6e588a8cc6cea9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:48.642066 20794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae45f0f92b3943bd92ec8eaa9270d0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:48.642066 20907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae45f0f92b3943bd92ec8eaa9270d0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:48.648625 21008 tablet_bootstrap.cc:492] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: Bootstrap starting.
I20260929 18:39:48.649443 21009 tablet_bootstrap.cc:492] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Bootstrap starting.
I20260929 18:39:48.649869 21010 tablet_bootstrap.cc:492] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Bootstrap starting.
I20260929 18:39:48.651036 21008 tablet_bootstrap.cc:654] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:48.651928 21008 log.cc:824] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:48.652122 21010 tablet_bootstrap.cc:654] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:48.652906 21010 log.cc:824] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:48.654145 21008 tablet_bootstrap.cc:492] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: No bootstrap required, opened a new log
I20260929 18:39:48.654366 21008 ts_tablet_manager.cc:1397] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:39:48.654949 21010 tablet_bootstrap.cc:492] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: No bootstrap required, opened a new log
I20260929 18:39:48.655146 21010 ts_tablet_manager.cc:1397] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:48.655232 21009 tablet_bootstrap.cc:654] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:48.655921 21009 log.cc:824] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:48.657598 21009 tablet_bootstrap.cc:492] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: No bootstrap required, opened a new log
I20260929 18:39:48.657779 21009 ts_tablet_manager.cc:1397] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260929 18:39:48.658775 21008 raft_consensus.cc:348] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.659070 21008 raft_consensus.cc:374] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:48.659157 21008 raft_consensus.cc:729] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555387434b2c4853a4f7914ae18a6328, State: Initialized, Role: FOLLOWER
I20260929 18:39:48.659258 21010 raft_consensus.cc:348] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.659497 21010 raft_consensus.cc:374] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:48.659588 21010 raft_consensus.cc:729] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362c5086c3df497c8955f4f4a8414055, State: Initialized, Role: FOLLOWER
I20260929 18:39:48.659653 21008 consensus_queue.cc:260] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.660048 21010 consensus_queue.cc:260] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.660866 21008 ts_tablet_manager.cc:1428] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:39:48.661002 21010 ts_tablet_manager.cc:1428] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:39:48.661235 20991 heartbeater.cc:503] Master 127.0.198.62:44141 was elected leader, sending a full tablet report...
I20260929 18:39:48.661269 20729 heartbeater.cc:503] Master 127.0.198.62:44141 was elected leader, sending a full tablet report...
I20260929 18:39:48.661931 21009 raft_consensus.cc:348] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.662242 21009 raft_consensus.cc:374] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:48.662315 21009 raft_consensus.cc:729] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982ff88d9dde4a518e798a28330a0ea2, State: Initialized, Role: FOLLOWER
I20260929 18:39:48.662729 21009 consensus_queue.cc:260] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.663321 20860 heartbeater.cc:503] Master 127.0.198.62:44141 was elected leader, sending a full tablet report...
I20260929 18:39:48.663682 21009 ts_tablet_manager.cc:1428] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 18:39:48.720693 20730 tablet.cc:2369] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:39:48.842649 20992 tablet.cc:2369] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:48.899142 21015 raft_consensus.cc:482] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:48.899341 21015 raft_consensus.cc:504] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.900966 21015 leader_election.cc:288] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555387434b2c4853a4f7914ae18a6328 (127.0.198.1:37293), 982ff88d9dde4a518e798a28330a0ea2 (127.0.198.2:42325)
I20260929 18:39:48.904516 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.900791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.2:42325 (local address 127.0.198.3:44731)
0929 18:39:48.900920 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.900937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.901032 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.901469 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.901472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.901492 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:48.901627 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.901635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.902623 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.902626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.903455 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.903464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.904121 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.904125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.904224 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.904246 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.904314 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.904373 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:39:48.904556 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.901091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:44731 (local address 127.0.198.2:42325)
0929 18:39:48.901198 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.901201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.901213 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:39:48.901270 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.901273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.901323 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.901415 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:48.901818 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.902506 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.903585 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.904005 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.904084 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.904137 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.904342 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.904423 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:39:48.904599 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.900946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.198.3:42065 (local address 127.0.198.1:37293)
0929 18:39:48.901076 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.901080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.901581 (+ 501us) server_negotiation.cc:407] Connection header received
0929 18:39:48.901645 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.901648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.901701 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.901776 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:48.902111 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.902627 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.903660 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.904083 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.904198 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.904256 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.904418 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.904493 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:39:48.904656 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.900791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37293 (local address 127.0.198.3:42065)
0929 18:39:48.901458 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.901472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:48.901573 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:48.901818 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:48.901821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:48.901841 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:48.901963 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.901999 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.902737 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.902740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.903539 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:48.903550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.904205 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.904208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:48.904297 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.904319 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:39:48.904392 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:39:48.904450 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":306,"thread_start_us":126,"threads_started":1}
I20260929 18:39:48.905332 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45f0f92b3943bd92ec8eaa9270d0fb" candidate_uuid: "362c5086c3df497c8955f4f4a8414055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555387434b2c4853a4f7914ae18a6328" is_pre_election: true
I20260929 18:39:48.905321 20814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45f0f92b3943bd92ec8eaa9270d0fb" candidate_uuid: "362c5086c3df497c8955f4f4a8414055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982ff88d9dde4a518e798a28330a0ea2" is_pre_election: true
I20260929 18:39:48.905623 20683 raft_consensus.cc:2454] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362c5086c3df497c8955f4f4a8414055 in term 0.
I20260929 18:39:48.905625 20814 raft_consensus.cc:2454] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362c5086c3df497c8955f4f4a8414055 in term 0.
W20260929 18:39:48.905634 20861 tablet.cc:2369] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:48.906173 20878 leader_election.cc:302] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 362c5086c3df497c8955f4f4a8414055, 555387434b2c4853a4f7914ae18a6328, 982ff88d9dde4a518e798a28330a0ea2; no voters:
I20260929 18:39:48.906417 21015 raft_consensus.cc:2781] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:48.906492 21015 raft_consensus.cc:482] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:48.906543 21015 raft_consensus.cc:3037] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:48.907534 21015 raft_consensus.cc:504] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.907886 21015 leader_election.cc:288] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [CANDIDATE]: Term 1 election: Requested vote from peers 555387434b2c4853a4f7914ae18a6328 (127.0.198.1:37293), 982ff88d9dde4a518e798a28330a0ea2 (127.0.198.2:42325)
I20260929 18:39:48.908234 20814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45f0f92b3943bd92ec8eaa9270d0fb" candidate_uuid: "362c5086c3df497c8955f4f4a8414055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982ff88d9dde4a518e798a28330a0ea2"
I20260929 18:39:48.908345 20814 raft_consensus.cc:3037] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:48.908344 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45f0f92b3943bd92ec8eaa9270d0fb" candidate_uuid: "362c5086c3df497c8955f4f4a8414055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555387434b2c4853a4f7914ae18a6328"
I20260929 18:39:48.908468 20683 raft_consensus.cc:3037] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:48.909358 20814 raft_consensus.cc:2454] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362c5086c3df497c8955f4f4a8414055 in term 1.
I20260929 18:39:48.909446 20683 raft_consensus.cc:2454] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362c5086c3df497c8955f4f4a8414055 in term 1.
I20260929 18:39:48.909678 20878 leader_election.cc:302] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c5086c3df497c8955f4f4a8414055, 982ff88d9dde4a518e798a28330a0ea2; no voters:
I20260929 18:39:48.909859 21015 raft_consensus.cc:2781] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:48.910115 21015 raft_consensus.cc:686] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 1 LEADER]: Becoming Leader. State: Replica: 362c5086c3df497c8955f4f4a8414055, State: Running, Role: LEADER
I20260929 18:39:48.910427 21015 consensus_queue.cc:237] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } }
I20260929 18:39:48.913551 20541 catalog_manager.cc:5495] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 362c5086c3df497c8955f4f4a8414055 (127.0.198.3). New cstate: current_term: 1 leader_uuid: "362c5086c3df497c8955f4f4a8414055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "362c5086c3df497c8955f4f4a8414055" member_type: VOTER last_known_addr { host: "127.0.198.3" port: 40343 } health_report { overall_health: HEALTHY } } }
I20260929 18:39:48.930240 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.926589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.0.198.3:40343)
0929 18:39:48.926709 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:48.926713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:48.926725 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:39:48.926799 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:48.926802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:48.926857 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:48.926949 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:48.927286 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.927823 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.928765 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:48.928999 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:48.929089 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:48.929390 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:48.929834 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:48.929933 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:48.930042 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:39:48.930103 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:39:48.936039 20683 raft_consensus.cc:1261] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328 [term 1 FOLLOWER]: Refusing update from remote peer 362c5086c3df497c8955f4f4a8414055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:48.936039 20814 raft_consensus.cc:1261] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 1 FOLLOWER]: Refusing update from remote peer 362c5086c3df497c8955f4f4a8414055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:48.936738 21018 consensus_queue.cc:1035] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982ff88d9dde4a518e798a28330a0ea2" member_type: VOTER last_known_addr { host: "127.0.198.2" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:48.936908 21015 consensus_queue.cc:1035] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555387434b2c4853a4f7914ae18a6328" member_type: VOTER last_known_addr { host: "127.0.198.1" port: 37293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:48.945369 21022 mvcc.cc:205] Tried to move back new op lower bound from 7334736645875134464 to 7334736645779873792. Current Snapshot: MvccSnapshot[applied={T|T < 7334736645875134464}]
I20260929 18:39:48.946043 21023 mvcc.cc:205] Tried to move back new op lower bound from 7334736645875134464 to 7334736645779873792. Current Snapshot: MvccSnapshot[applied={T|T < 7334736645875134464}]
I20260929 18:39:48.954242 20539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:48.954412 20539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:48.956640 20539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b240bf8530bd4979a7fecf9bf86aef38: Sending DeleteTablet for 3 replicas of tablet ae45f0f92b3943bd92ec8eaa9270d0fb
I20260929 18:39:48.957439 20663 tablet_service.cc:1514] Processing DeleteTablet for tablet ae45f0f92b3943bd92ec8eaa9270d0fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:48 UTC) from {username='slave'} at 127.0.0.1:59456
I20260929 18:39:48.957527 20794 tablet_service.cc:1514] Processing DeleteTablet for tablet ae45f0f92b3943bd92ec8eaa9270d0fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:48 UTC) from {username='slave'} at 127.0.0.1:46092
I20260929 18:39:48.958083 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 20601
I20260929 18:39:48.958091 20907 tablet_service.cc:1514] Processing DeleteTablet for tablet ae45f0f92b3943bd92ec8eaa9270d0fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:48 UTC) from {username='slave'} at 127.0.0.1:33802
I20260929 18:39:48.958220 21031 tablet_replica.cc:330] T ae45f0f92b3943bd92ec8eaa9270d0fb P 555387434b2c4853a4f7914ae18a6328: stopping tablet replica
I20260929 18:39:48.958732 21032 tablet_replica.cc:330] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: stopping tablet replica
I20260929 18:39:48.959031 21032 raft_consensus.cc:2229] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:48.959630 21032 raft_consensus.cc:2258] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:48.961397 21032 ts_tablet_manager.cc:1905] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:48.962175 21033 tablet_replica.cc:330] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: stopping tablet replica
I20260929 18:39:48.962517 21033 raft_consensus.cc:2229] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:39:48.964334 21032 ts_tablet_manager.cc:1918] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:48.964457 21032 log.cc:1196] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-1/wal/wals/ae45f0f92b3943bd92ec8eaa9270d0fb
I20260929 18:39:48.964774 21032 ts_tablet_manager.cc:1939] T ae45f0f92b3943bd92ec8eaa9270d0fb P 982ff88d9dde4a518e798a28330a0ea2: Deleting consensus metadata
W20260929 18:39:48.965454 20526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.198.1:37293 (error 108)
I20260929 18:39:48.966078 20525 catalog_manager.cc:4841] TS 982ff88d9dde4a518e798a28330a0ea2 (127.0.198.2:42325): tablet ae45f0f92b3943bd92ec8eaa9270d0fb (table test-table [id=824497d7372b48a2a6e588a8cc6cea9c]) successfully deleted
I20260929 18:39:48.966266 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 20732
I20260929 18:39:48.963008 21033 raft_consensus.cc:2258] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:48.968077 21033 ts_tablet_manager.cc:1905] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:48.970665 21033 ts_tablet_manager.cc:1918] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:48.970762 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:48.970491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.1:37293 (local address 127.0.0.1:59472)
0929 18:39:48.970608 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:48.970683 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:39:48.970816 21033 log.cc:1196] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Deleting WAL directory at /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706942746699-792-0/minicluster-data/ts-2/wal/wals/ae45f0f92b3943bd92ec8eaa9270d0fb
W20260929 18:39:48.971035 20526 catalog_manager.cc:4584] TS 555387434b2c4853a4f7914ae18a6328 (127.0.198.1:37293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae45f0f92b3943bd92ec8eaa9270d0fb: Network error: Client connection negotiation failed: client connection to 127.0.198.1:37293: connect: Connection refused (error 111)
I20260929 18:39:48.971107 21033 ts_tablet_manager.cc:1939] T ae45f0f92b3943bd92ec8eaa9270d0fb P 362c5086c3df497c8955f4f4a8414055: Deleting consensus metadata
I20260929 18:39:48.972246 20525 catalog_manager.cc:4841] TS 362c5086c3df497c8955f4f4a8414055 (127.0.198.3:40343): tablet ae45f0f92b3943bd92ec8eaa9270d0fb (table test-table [id=824497d7372b48a2a6e588a8cc6cea9c]) successfully deleted
I20260929 18:39:48.973148 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 20863
I20260929 18:39:48.979705 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 20510
2026-09-29T18:39:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1207 ms)
[----------] 6 tests from EncryptionPolicyTest (10224 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:49 dist-test-slave-q5qx krb5kdc[21037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:49 dist-test-slave-q5qx krb5kdc[21037](info): set up 2 sockets
Sep 29 18:39:49 dist-test-slave-q5qx krb5kdc[21037](info): commencing operation
krb5kdc: starting...
W20260929 18:39:49.945921 792 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:50 dist-test-slave-q5qx krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:50.065706 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:50.173405 21050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:50.173666 21050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:50.173715 21050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:50.177688 21050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:50.177752 21050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:50.177768 21050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:50.177785 21050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:50.177798 21050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:50.183141 21050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:50.184144 21050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:50.185153 21050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:50.193486 21055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:50.193549 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:50.193615 21050 server_base.cc:1034] running on GCE node
W20260929 18:39:50.193538 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:50.194403 21050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:50.195510 21050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:50.196704 21050 hybrid_clock.cc:648] HybridClock initialized: now 1790707190196680 us; error 28 us; skew 500 ppm
Sep 29 18:39:50 dist-test-slave-q5qx krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:50.199607 21050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:50.200591 21050 webserver.cc:460] Webserver started at http://127.0.0.1:40471/ using document root <none> and password file <none>
I20260929 18:39:50.201133 21050 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:50.201197 21050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:50.201395 21050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:50.203478 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "1d380b5732b742ab98de41d27fb5ec78"
format_stamp: "Formatted at 2026-09-29 18:39:50 on dist-test-slave-q5qx"
server_key: "9e8d9bf149329727f3105b3c6193dca2"
server_key_iv: "1123051bc24d4b831eadbc4e9bab1b73"
server_key_version: "encryptionkey@0"
I20260929 18:39:50.203998 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "1d380b5732b742ab98de41d27fb5ec78"
format_stamp: "Formatted at 2026-09-29 18:39:50 on dist-test-slave-q5qx"
server_key: "9e8d9bf149329727f3105b3c6193dca2"
server_key_iv: "1123051bc24d4b831eadbc4e9bab1b73"
server_key_version: "encryptionkey@0"
I20260929 18:39:50.207648 21050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:39:50.210161 21065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:50.211520 21050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:50.211681 21050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "1d380b5732b742ab98de41d27fb5ec78"
format_stamp: "Formatted at 2026-09-29 18:39:50 on dist-test-slave-q5qx"
server_key: "9e8d9bf149329727f3105b3c6193dca2"
server_key_iv: "1123051bc24d4b831eadbc4e9bab1b73"
server_key_version: "encryptionkey@0"
I20260929 18:39:50.211787 21050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:50.231355 21050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:50.234586 21050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:50.234776 21050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:50.241487 21050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43893
I20260929 18:39:50.241583 21117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43893 every 8 connection(s)
I20260929 18:39:50.242476 21050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:50.245244 21118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:50.251286 21118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78: Bootstrap starting.
I20260929 18:39:50.251864 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21050
I20260929 18:39:50.252003 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:50.252302 792 external_mini_cluster.cc:1428] Setting key b4a7b1db6318bd0dd93a71164bb9f688
I20260929 18:39:50.254364 21118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:50.255101 21118 log.cc:824] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:50.257704 21118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78: No bootstrap required, opened a new log
I20260929 18:39:50.261925 21118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d380b5732b742ab98de41d27fb5ec78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43893 } }
I20260929 18:39:50.262228 21118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:50.262319 21118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d380b5732b742ab98de41d27fb5ec78, State: Initialized, Role: FOLLOWER
I20260929 18:39:50.262773 21118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d380b5732b742ab98de41d27fb5ec78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43893 } }
I20260929 18:39:50.262897 21118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:50.262961 21118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:50.263078 21118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:50.264006 21118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d380b5732b742ab98de41d27fb5ec78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43893 } }
I20260929 18:39:50.264348 21118 leader_election.cc:302] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d380b5732b742ab98de41d27fb5ec78; no voters:
I20260929 18:39:50.264681 21118 leader_election.cc:288] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:50.264802 21123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:50.265028 21123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [term 1 LEADER]: Becoming Leader. State: Replica: 1d380b5732b742ab98de41d27fb5ec78, State: Running, Role: LEADER
I20260929 18:39:50.265407 21123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d380b5732b742ab98de41d27fb5ec78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43893 } }
I20260929 18:39:50.265945 21118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:50.267313 21125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d380b5732b742ab98de41d27fb5ec78. Latest consensus state: current_term: 1 leader_uuid: "1d380b5732b742ab98de41d27fb5ec78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d380b5732b742ab98de41d27fb5ec78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43893 } } }
I20260929 18:39:50.267364 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d380b5732b742ab98de41d27fb5ec78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d380b5732b742ab98de41d27fb5ec78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43893 } } }
I20260929 18:39:50.267444 21125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:50.267460 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78 [sys.catalog]: This master's current role is: LEADER
Sep 29 18:39:50 dist-test-slave-q5qx krb5kdc[21037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:39:50.267886 21135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:50.271211 21135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:50.274123 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:50.253803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57043 (local address 127.0.0.1:43893)
0929 18:39:50.254348 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:50.254364 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:50.254400 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:39:50.255114 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:50.255133 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:50.255554 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:50.255933 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:50.256657 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:50.258170 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:50.259164 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:50.260030 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:50.260203 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:50.268362 (+ 8159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:50.268369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:50.268388 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:50.268424 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:50.270533 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:50.271221 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:50.271227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:50.271233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:50.271373 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:50.271769 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:50.271772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:50.271774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:50.272296 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:50.272502 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:50.272859 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 18:39:50.273109 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":179,"threads_started":1}
I20260929 18:39:50.277271 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:50.278865 21135 catalog_manager.cc:1347] Generated new cluster ID: cd375b6422ee41ffbd0b3664d4631497
I20260929 18:39:50.278928 21135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:50.286589 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:50.279448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44769 (local address 127.0.0.1:43893)
0929 18:39:50.279563 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:50.279567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:50.279779 (+ 212us) server_negotiation.cc:407] Connection header received
0929 18:39:50.279881 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:50.279884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:50.279926 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:50.280002 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:50.280720 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:50.281234 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:50.282021 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:50.282315 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:50.282404 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:50.283710 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:50.283712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:50.283714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:50.283736 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:50.284972 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:50.285403 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:50.285405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:50.285407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:50.285476 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:50.285759 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:50.285762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:50.285763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:50.285923 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:50.286087 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:50.286334 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:39:50.286423 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:39:50.305377 21135 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:50.306396 21135 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:50.318384 21135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d380b5732b742ab98de41d27fb5ec78: Generated new TSK 0
I20260929 18:39:50.319268 21135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:50.332651 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21050
W20260929 18:39:50.337495 21035 connection.cc:535] client connection to 127.0.0.1:43893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43893): Connection reset by peer (error 104)
2026-09-29T18:39:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:50 dist-test-slave-q5qx krb5kdc[21149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:50 dist-test-slave-q5qx krb5kdc[21149](info): set up 2 sockets
Sep 29 18:39:50 dist-test-slave-q5qx krb5kdc[21149](info): commencing operation
krb5kdc: starting...
W20260929 18:39:51.297885 792 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:51 dist-test-slave-q5qx krb5kdc[21149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:51.419212 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:51.528657 21162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:51.528935 21162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:51.528995 21162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:51.532991 21162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:51.533054 21162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:51.533082 21162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:51.533102 21162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:51.533121 21162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:51.538502 21162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:51.539528 21162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:51.540513 21162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:51.549069 21168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:51.549077 21170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:51.549248 21162 server_base.cc:1034] running on GCE node
W20260929 18:39:51.549157 21167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:51.549933 21162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:51.551126 21162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:51.552315 21162 hybrid_clock.cc:648] HybridClock initialized: now 1790707191552279 us; error 37 us; skew 500 ppm
Sep 29 18:39:51 dist-test-slave-q5qx krb5kdc[21149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:51.555249 21162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:51.556201 21162 webserver.cc:460] Webserver started at http://127.0.0.1:43129/ using document root <none> and password file <none>
I20260929 18:39:51.556734 21162 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:51.556800 21162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:51.557016 21162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:51.559134 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "3768744904fc4fe7bfbca8ac265263b0"
format_stamp: "Formatted at 2026-09-29 18:39:51 on dist-test-slave-q5qx"
server_key: "a944f83edbe9aec9002be0f1f263ccf0"
server_key_iv: "637d924d85f469c595a0c8aa696d01a7"
server_key_version: "encryptionkey@0"
I20260929 18:39:51.559687 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "3768744904fc4fe7bfbca8ac265263b0"
format_stamp: "Formatted at 2026-09-29 18:39:51 on dist-test-slave-q5qx"
server_key: "a944f83edbe9aec9002be0f1f263ccf0"
server_key_iv: "637d924d85f469c595a0c8aa696d01a7"
server_key_version: "encryptionkey@0"
I20260929 18:39:51.563441 21162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:51.566112 21177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:51.567405 21162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:51.567525 21162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "3768744904fc4fe7bfbca8ac265263b0"
format_stamp: "Formatted at 2026-09-29 18:39:51 on dist-test-slave-q5qx"
server_key: "a944f83edbe9aec9002be0f1f263ccf0"
server_key_iv: "637d924d85f469c595a0c8aa696d01a7"
server_key_version: "encryptionkey@0"
I20260929 18:39:51.567642 21162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:51.623202 21162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:51.626387 21162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:51.626566 21162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:51.633327 21162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35773
I20260929 18:39:51.633396 21229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35773 every 8 connection(s)
I20260929 18:39:51.634305 21162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:51.636068 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21162
I20260929 18:39:51.636216 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:51.636503 792 external_mini_cluster.cc:1428] Setting key 836ed214f1c384e32a01cadbd849e6da
I20260929 18:39:51.637287 21230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:51.643471 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0: Bootstrap starting.
I20260929 18:39:51.645138 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:51.637788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47903 (local address 127.0.0.1:35773)
0929 18:39:51.638190 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:51.638206 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:51.638241 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:39:51.638875 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:51.638890 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:51.639236 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:51.639512 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:51.639841 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:51.640804 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:51.641953 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:51.642465 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:51.642577 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:51.643458 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:51.643463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:51.643475 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:51.643895 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:51.643949 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:51.643973 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:51.644071 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:51.644256 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 18:39:51.644385 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":138,"threads_started":1}
I20260929 18:39:51.645727 21230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:51.646553 21230 log.cc:824] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:51.648216 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:51.648836 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0: No bootstrap required, opened a new log
I20260929 18:39:51.653152 21230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3768744904fc4fe7bfbca8ac265263b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260929 18:39:51.653390 21230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:51.653475 21230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3768744904fc4fe7bfbca8ac265263b0, State: Initialized, Role: FOLLOWER
I20260929 18:39:51.653932 21230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3768744904fc4fe7bfbca8ac265263b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260929 18:39:51.654124 21230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:51.654224 21230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:51.654331 21230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:51.654697 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:51.650449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40559 (local address 127.0.0.1:35773)
0929 18:39:51.650571 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:51.650575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:51.650729 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:39:51.650839 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:51.650843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:51.650940 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:51.651025 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:51.651438 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:51.651949 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:51.653064 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:51.653304 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:51.653392 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:51.654263 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:51.654266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:51.654268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:51.654279 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:51.654305 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:51.654318 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:51.654409 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:51.654524 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:39:51.654555 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:39:51.655489 21230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3768744904fc4fe7bfbca8ac265263b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260929 18:39:51.655825 21230 leader_election.cc:302] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3768744904fc4fe7bfbca8ac265263b0; no voters:
I20260929 18:39:51.656100 21230 leader_election.cc:288] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:51.656200 21240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:51.656412 21240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3768744904fc4fe7bfbca8ac265263b0, State: Running, Role: LEADER
I20260929 18:39:51.656739 21240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3768744904fc4fe7bfbca8ac265263b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260929 18:39:51.657230 21230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:51.658241 21241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3768744904fc4fe7bfbca8ac265263b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3768744904fc4fe7bfbca8ac265263b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } } }
I20260929 18:39:51.658366 21241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:51.658754 21249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:51.658718 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3768744904fc4fe7bfbca8ac265263b0. Latest consensus state: current_term: 1 leader_uuid: "3768744904fc4fe7bfbca8ac265263b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3768744904fc4fe7bfbca8ac265263b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } } }
I20260929 18:39:51.658859 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:51.661731 21249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:51.666966 21249 catalog_manager.cc:1347] Generated new cluster ID: b9449e4ba976417caa2e32201f045d1a
I20260929 18:39:51.667037 21249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:51.688525 21249 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:51.689498 21249 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:51.696448 21249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3768744904fc4fe7bfbca8ac265263b0: Generated new TSK 0
I20260929 18:39:51.697139 21249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:51.706537 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21162
W20260929 18:39:51.711115 21147 connection.cc:535] client connection to 127.0.0.1:35773 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35773): Connection reset by peer (error 104)
2026-09-29T18:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:51 dist-test-slave-q5qx krb5kdc[21261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:51 dist-test-slave-q5qx krb5kdc[21261](info): set up 2 sockets
Sep 29 18:39:51 dist-test-slave-q5qx krb5kdc[21261](info): commencing operation
krb5kdc: starting...
W20260929 18:39:52.665889 792 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:52 dist-test-slave-q5qx krb5kdc[21261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:52.785388 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:52.894886 21274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:52.895247 21274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:52.895321 21274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:52.899403 21274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:52.899477 21274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:52.899502 21274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:52.899525 21274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:52.899554 21274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:52.905001 21274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:52.906121 21274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:52.907141 21274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:52.915373 21280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.915369 21282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.915369 21279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:52.915436 21274 server_base.cc:1034] running on GCE node
I20260929 18:39:52.916097 21274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:52.917222 21274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:52.918393 21274 hybrid_clock.cc:648] HybridClock initialized: now 1790707192918371 us; error 37 us; skew 500 ppm
Sep 29 18:39:52 dist-test-slave-q5qx krb5kdc[21261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:52.921242 21274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:52.922200 21274 webserver.cc:460] Webserver started at http://127.0.0.1:42283/ using document root <none> and password file <none>
I20260929 18:39:52.922737 21274 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:52.922803 21274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:52.923025 21274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:52.925016 21274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "cef8e683cdf74e27a5b42a7288773402"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-q5qx"
server_key: "dfa24a37d773c32269fae08e59163fa5"
server_key_iv: "16bc5b22f6e7760cbfe8653330270ca5"
server_key_version: "encryptionkey@0"
I20260929 18:39:52.925544 21274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "cef8e683cdf74e27a5b42a7288773402"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-q5qx"
server_key: "dfa24a37d773c32269fae08e59163fa5"
server_key_iv: "16bc5b22f6e7760cbfe8653330270ca5"
server_key_version: "encryptionkey@0"
I20260929 18:39:52.929486 21274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:39:52.932214 21289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.933596 21274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:52.933734 21274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "cef8e683cdf74e27a5b42a7288773402"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-q5qx"
server_key: "dfa24a37d773c32269fae08e59163fa5"
server_key_iv: "16bc5b22f6e7760cbfe8653330270ca5"
server_key_version: "encryptionkey@0"
I20260929 18:39:52.933843 21274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:52.938431 21274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:52.941565 21274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:52.941726 21274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:52.948195 21274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44091
I20260929 18:39:52.948272 21341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44091 every 8 connection(s)
I20260929 18:39:52.949128 21274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:52.951380 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21274
I20260929 18:39:52.951509 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:52.951786 792 external_mini_cluster.cc:1428] Setting key f588601dfd59e90843d0caa4733c158f
I20260929 18:39:52.951967 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:52.957450 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.953179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53657 (local address 127.0.0.1:44091)
0929 18:39:52.953605 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:52.953624 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:52.953665 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:39:52.954578 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:52.954602 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:52.954958 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:52.955359 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:52.956036 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:52.956042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:52.956056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:52.956068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:52.956129 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:52.956153 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:52.956281 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:52.956403 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:39:52.956561 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260929 18:39:52.959069 21342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402: Bootstrap starting.
I20260929 18:39:52.960326 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:52.961297 21342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:52.962005 21342 log.cc:824] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:52.963801 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.962211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40887 (local address 127.0.0.1:44091)
0929 18:39:52.962325 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:52.962329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:52.962631 (+ 302us) server_negotiation.cc:407] Connection header received
0929 18:39:52.962712 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:52.962716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:52.962749 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:52.962814 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:52.963459 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:52.963462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:52.963463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:52.963473 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:52.963506 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:52.963516 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:52.963579 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:52.963672 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 18:39:52.963701 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:39:52.964227 21342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402: No bootstrap required, opened a new log
I20260929 18:39:52.967669 21342 raft_consensus.cc:348] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef8e683cdf74e27a5b42a7288773402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260929 18:39:52.967898 21342 raft_consensus.cc:374] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:52.967957 21342 raft_consensus.cc:729] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef8e683cdf74e27a5b42a7288773402, State: Initialized, Role: FOLLOWER
I20260929 18:39:52.968397 21342 consensus_queue.cc:260] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef8e683cdf74e27a5b42a7288773402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260929 18:39:52.968525 21342 raft_consensus.cc:388] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:52.968588 21342 raft_consensus.cc:482] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:52.968681 21342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:52.969573 21342 raft_consensus.cc:504] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef8e683cdf74e27a5b42a7288773402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260929 18:39:52.969894 21342 leader_election.cc:302] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cef8e683cdf74e27a5b42a7288773402; no voters:
I20260929 18:39:52.970209 21342 leader_election.cc:288] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:52.970325 21352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:52.970546 21352 raft_consensus.cc:686] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [term 1 LEADER]: Becoming Leader. State: Replica: cef8e683cdf74e27a5b42a7288773402, State: Running, Role: LEADER
I20260929 18:39:52.970875 21352 consensus_queue.cc:237] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef8e683cdf74e27a5b42a7288773402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260929 18:39:52.971346 21342 sys_catalog.cc:564] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:52.972400 21353 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cef8e683cdf74e27a5b42a7288773402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef8e683cdf74e27a5b42a7288773402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } } }
I20260929 18:39:52.972530 21353 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:52.972867 21354 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cef8e683cdf74e27a5b42a7288773402. Latest consensus state: current_term: 1 leader_uuid: "cef8e683cdf74e27a5b42a7288773402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef8e683cdf74e27a5b42a7288773402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } } }
I20260929 18:39:52.972947 21354 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:52.973255 21361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:52.976421 21361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:52.981848 21361 catalog_manager.cc:1347] Generated new cluster ID: e294366796634e5aacd82809406ab664
I20260929 18:39:52.981923 21361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:53.000645 21361 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:53.001032 21361 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:53.019265 21361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cef8e683cdf74e27a5b42a7288773402: Generated new TSK 0
I20260929 18:39:53.020179 21361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:53.029600 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21274
2026-09-29T18:39:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:53 dist-test-slave-q5qx krb5kdc[21373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:53 dist-test-slave-q5qx krb5kdc[21373](info): set up 2 sockets
Sep 29 18:39:53 dist-test-slave-q5qx krb5kdc[21373](info): commencing operation
krb5kdc: starting...
W20260929 18:39:53.986092 792 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:54 dist-test-slave-q5qx krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:54.109740 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:54.218312 21386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:54.218596 21386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:54.218653 21386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:54.222875 21386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:54.222942 21386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:54.222986 21386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:54.223042 21386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:54.223068 21386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:54.228615 21386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:54.229720 21386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:54.230815 21386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:54.239603 21391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:54.239722 21392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:54.239760 21386 server_base.cc:1034] running on GCE node
W20260929 18:39:54.239609 21394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:54.240658 21386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:54.241729 21386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:54.242897 21386 hybrid_clock.cc:648] HybridClock initialized: now 1790707194242869 us; error 40 us; skew 500 ppm
Sep 29 18:39:54 dist-test-slave-q5qx krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:54.251065 21386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:54.252007 21386 webserver.cc:460] Webserver started at http://127.0.0.1:44055/ using document root <none> and password file <none>
I20260929 18:39:54.252622 21386 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:54.252704 21386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:54.252924 21386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:54.255010 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "06bcb3243bcc4a5aa2a7063b67f980e9"
format_stamp: "Formatted at 2026-09-29 18:39:54 on dist-test-slave-q5qx"
server_key: "7258975310e69a8ae900bea339317cea"
server_key_iv: "294bb6fbb67709c04bc650df21f997f4"
server_key_version: "encryptionkey@0"
I20260929 18:39:54.255558 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "06bcb3243bcc4a5aa2a7063b67f980e9"
format_stamp: "Formatted at 2026-09-29 18:39:54 on dist-test-slave-q5qx"
server_key: "7258975310e69a8ae900bea339317cea"
server_key_iv: "294bb6fbb67709c04bc650df21f997f4"
server_key_version: "encryptionkey@0"
I20260929 18:39:54.259351 21386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:39:54.261777 21401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:54.263175 21386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:39:54.263320 21386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "06bcb3243bcc4a5aa2a7063b67f980e9"
format_stamp: "Formatted at 2026-09-29 18:39:54 on dist-test-slave-q5qx"
server_key: "7258975310e69a8ae900bea339317cea"
server_key_iv: "294bb6fbb67709c04bc650df21f997f4"
server_key_version: "encryptionkey@0"
I20260929 18:39:54.263428 21386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:54.273818 21386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:54.276809 21386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:54.276995 21386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:54.283769 21386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34247
I20260929 18:39:54.283788 21453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34247 every 8 connection(s)
I20260929 18:39:54.284771 21386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:54.286129 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21386
I20260929 18:39:54.286326 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:54.286623 792 external_mini_cluster.cc:1428] Setting key 5872bd793accb0a0c32a9489131b56c0
I20260929 18:39:54.288130 21454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:54.293648 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9: Bootstrap starting.
Sep 29 18:39:54 dist-test-slave-q5qx krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:39:54.296073 21454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:54.296870 21454 log.cc:824] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:54.299168 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9: No bootstrap required, opened a new log
I20260929 18:39:54.300237 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:54.287984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37811 (local address 127.0.0.1:34247)
0929 18:39:54.288423 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:54.288438 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:54.288473 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:39:54.289110 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:54.289128 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:54.289480 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:54.289790 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:54.290600 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.291575 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.292415 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.292883 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.293021 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:54.295725 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:54.295731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:54.295745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:54.295773 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:54.297458 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.297961 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.297983 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.297989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.298114 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.298383 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.298385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.298386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.298766 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:54.298902 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:54.299196 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:39:54.299495 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260929 18:39:54.302304 21454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260929 18:39:54.302583 21454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:54.302646 21454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bcb3243bcc4a5aa2a7063b67f980e9, State: Initialized, Role: FOLLOWER
I20260929 18:39:54.303063 21454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260929 18:39:54.303188 21454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:54.303248 21454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:54.303344 21454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:54.303452 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:54.304515 21454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260929 18:39:54.304904 21454 leader_election.cc:302] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06bcb3243bcc4a5aa2a7063b67f980e9; no voters:
I20260929 18:39:54.305248 21454 leader_election.cc:288] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:54.305336 21463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:54.305588 21463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [term 1 LEADER]: Becoming Leader. State: Replica: 06bcb3243bcc4a5aa2a7063b67f980e9, State: Running, Role: LEADER
I20260929 18:39:54.306093 21463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260929 18:39:54.306594 21454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:54.307717 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06bcb3243bcc4a5aa2a7063b67f980e9. Latest consensus state: current_term: 1 leader_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } } }
I20260929 18:39:54.307823 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:54.308184 21473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:54.308148 21464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bcb3243bcc4a5aa2a7063b67f980e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } } }
I20260929 18:39:54.308224 21464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:54.311322 21473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:54.315949 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:54.305656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49961 (local address 127.0.0.1:34247)
0929 18:39:54.305762 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:54.305766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:54.306232 (+ 466us) server_negotiation.cc:407] Connection header received
0929 18:39:54.306423 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:54.306426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:54.306485 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:54.306565 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:54.308553 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.309309 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.310479 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.310804 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.310903 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:54.312485 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:54.312489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:54.312492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:54.312518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:54.314167 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.314659 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.314663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.314664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.314754 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.315019 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.315022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.315024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.315249 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:54.315369 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:54.315669 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 18:39:54.315787 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:39:54.319324 21473 catalog_manager.cc:1347] Generated new cluster ID: 82303e07a6484b8db68e23f1368c2e63
I20260929 18:39:54.319391 21473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:54.329358 21473 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:54.330684 21473 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:54.337071 21473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06bcb3243bcc4a5aa2a7063b67f980e9: Generated new TSK 0
I20260929 18:39:54.337875 21473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:54.348407 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21386
2026-09-29T18:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:54 dist-test-slave-q5qx krb5kdc[21485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:54 dist-test-slave-q5qx krb5kdc[21485](info): set up 2 sockets
Sep 29 18:39:54 dist-test-slave-q5qx krb5kdc[21485](info): commencing operation
krb5kdc: starting...
W20260929 18:39:55.294175 792 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:55 dist-test-slave-q5qx krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:55.418473 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:55.527607 21498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:55.527899 21498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:55.527956 21498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:55.532121 21498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:55.532197 21498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:55.532223 21498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:55.532244 21498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:55.532276 21498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:55.537747 21498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:55.538852 21498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:55.539880 21498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:55.548400 21503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:55.548549 21498 server_base.cc:1034] running on GCE node
W20260929 18:39:55.548403 21504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:55.548394 21506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:55.549297 21498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:55.550418 21498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:55.551568 21498 hybrid_clock.cc:648] HybridClock initialized: now 1790707195551554 us; error 28 us; skew 500 ppm
Sep 29 18:39:55 dist-test-slave-q5qx krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:55.554272 21498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:55.555136 21498 webserver.cc:460] Webserver started at http://127.0.0.1:32923/ using document root <none> and password file <none>
I20260929 18:39:55.555680 21498 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:55.555743 21498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:55.555922 21498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:55.557942 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "b4ed90570eb34e4d9025bf42166a39c5"
format_stamp: "Formatted at 2026-09-29 18:39:55 on dist-test-slave-q5qx"
server_key: "1554ffda8d0f46ba740e37acccfcd151"
server_key_iv: "d6a7e0503c92d58a4bd9377bc809afcf"
server_key_version: "encryptionkey@0"
I20260929 18:39:55.558540 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "b4ed90570eb34e4d9025bf42166a39c5"
format_stamp: "Formatted at 2026-09-29 18:39:55 on dist-test-slave-q5qx"
server_key: "1554ffda8d0f46ba740e37acccfcd151"
server_key_iv: "d6a7e0503c92d58a4bd9377bc809afcf"
server_key_version: "encryptionkey@0"
I20260929 18:39:55.562251 21498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:55.564704 21513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:55.565904 21498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:55.566057 21498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "b4ed90570eb34e4d9025bf42166a39c5"
format_stamp: "Formatted at 2026-09-29 18:39:55 on dist-test-slave-q5qx"
server_key: "1554ffda8d0f46ba740e37acccfcd151"
server_key_iv: "d6a7e0503c92d58a4bd9377bc809afcf"
server_key_version: "encryptionkey@0"
I20260929 18:39:55.566169 21498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:55.582906 21498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:55.586321 21498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:55.586536 21498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:55.593767 21498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34303
I20260929 18:39:55.593835 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34303 every 8 connection(s)
I20260929 18:39:55.594801 21498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:55.597602 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:55.603083 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5: Bootstrap starting.
I20260929 18:39:55.604665 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21498
I20260929 18:39:55.604794 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:55.605041 792 external_mini_cluster.cc:1428] Setting key 3f7ed5f0a7256c905e241d86e6d6fb7b
I20260929 18:39:55.605587 21566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:55.606416 21566 log.cc:824] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:55.608664 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5: No bootstrap required, opened a new log
I20260929 18:39:55.612927 21566 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ed90570eb34e4d9025bf42166a39c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34303 } }
I20260929 18:39:55.613224 21566 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:55.613284 21566 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ed90570eb34e4d9025bf42166a39c5, State: Initialized, Role: FOLLOWER
I20260929 18:39:55.613781 21566 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ed90570eb34e4d9025bf42166a39c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34303 } }
I20260929 18:39:55.613938 21566 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:55.614032 21566 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:55.614130 21566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:55.615023 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.606181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45547 (local address 127.0.0.1:34303)
0929 18:39:55.606649 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.606669 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.606710 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:39:55.607452 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.607475 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.607892 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.608266 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:55.608661 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.610167 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.611129 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.611864 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.612036 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.612853 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:55.612859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:55.612877 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:55.613407 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:55.613475 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:55.613506 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:55.613598 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.613774 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.614032 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260929 18:39:55.615279 21566 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ed90570eb34e4d9025bf42166a39c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34303 } }
I20260929 18:39:55.615641 21566 leader_election.cc:302] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ed90570eb34e4d9025bf42166a39c5; no voters:
I20260929 18:39:55.615988 21566 leader_election.cc:288] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:55.616094 21571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:55.616333 21571 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [term 1 LEADER]: Becoming Leader. State: Replica: b4ed90570eb34e4d9025bf42166a39c5, State: Running, Role: LEADER
I20260929 18:39:55.616643 21571 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ed90570eb34e4d9025bf42166a39c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34303 } }
I20260929 18:39:55.617189 21566 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:55.617844 21572 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ed90570eb34e4d9025bf42166a39c5. Latest consensus state: current_term: 1 leader_uuid: "b4ed90570eb34e4d9025bf42166a39c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ed90570eb34e4d9025bf42166a39c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34303 } } }
I20260929 18:39:55.617955 21572 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:55.618450 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:55.618610 21573 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ed90570eb34e4d9025bf42166a39c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ed90570eb34e4d9025bf42166a39c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34303 } } }
I20260929 18:39:55.618693 21573 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5 [sys.catalog]: This master's current role is: LEADER
W20260929 18:39:55.622244 21586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:39:55.622370 21586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:39:55.622462 21579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:55.623167 21579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:55.625159 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.620475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59675 (local address 127.0.0.1:34303)
0929 18:39:55.620586 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.620591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.620811 (+ 220us) server_negotiation.cc:407] Connection header received
0929 18:39:55.620977 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.620980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.621037 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.621083 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:55.621485 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.622256 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.623440 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.623778 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.623879 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.624725 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:55.624728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:55.624731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:55.624745 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:55.624774 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:55.624787 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:55.624860 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.624948 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.625001 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:39:55.629210 21579 catalog_manager.cc:1347] Generated new cluster ID: e595be77725e4cc096206371b700a0ff
I20260929 18:39:55.629279 21579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:55.643723 21579 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:55.644641 21579 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:55.657262 21579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ed90570eb34e4d9025bf42166a39c5: Generated new TSK 0
I20260929 18:39:55.657958 21579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:55.664400 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21498
2026-09-29T18:39:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:55 dist-test-slave-q5qx krb5kdc[21597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:55 dist-test-slave-q5qx krb5kdc[21597](info): set up 2 sockets
Sep 29 18:39:55 dist-test-slave-q5qx krb5kdc[21597](info): commencing operation
krb5kdc: starting...
W20260929 18:39:56.617940 792 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:56 dist-test-slave-q5qx krb5kdc[21597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:56.738504 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:56.846524 21610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:56.846817 21610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:56.846875 21610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:56.851022 21610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:56.851097 21610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:56.851125 21610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:56.851145 21610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:56.851173 21610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:56.856657 21610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:56.857750 21610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:56.858804 21610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:56.867161 21616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:56.867192 21618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:56.867381 21610 server_base.cc:1034] running on GCE node
W20260929 18:39:56.867290 21615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:56.868049 21610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:56.869146 21610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:56.870302 21610 hybrid_clock.cc:648] HybridClock initialized: now 1790707196870288 us; error 27 us; skew 500 ppm
Sep 29 18:39:56 dist-test-slave-q5qx krb5kdc[21597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:56.872856 21610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:56.873764 21610 webserver.cc:460] Webserver started at http://127.0.0.1:43391/ using document root <none> and password file <none>
I20260929 18:39:56.874349 21610 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:56.874416 21610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:56.874601 21610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:56.876610 21610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "fdcd1489837d41dca93795b3c357c7c9"
format_stamp: "Formatted at 2026-09-29 18:39:56 on dist-test-slave-q5qx"
server_key: "9c74684182104b164cb214f0ddee3d38"
server_key_iv: "6c3a031751c549f6db1b0935e30c140c"
server_key_version: "encryptionkey@0"
I20260929 18:39:56.877148 21610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "fdcd1489837d41dca93795b3c357c7c9"
format_stamp: "Formatted at 2026-09-29 18:39:56 on dist-test-slave-q5qx"
server_key: "9c74684182104b164cb214f0ddee3d38"
server_key_iv: "6c3a031751c549f6db1b0935e30c140c"
server_key_version: "encryptionkey@0"
I20260929 18:39:56.880829 21610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:39:56.883373 21625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:56.884716 21610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:39:56.884845 21610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "fdcd1489837d41dca93795b3c357c7c9"
format_stamp: "Formatted at 2026-09-29 18:39:56 on dist-test-slave-q5qx"
server_key: "9c74684182104b164cb214f0ddee3d38"
server_key_iv: "6c3a031751c549f6db1b0935e30c140c"
server_key_version: "encryptionkey@0"
I20260929 18:39:56.884946 21610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:56.889515 21610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:56.892556 21610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:56.892733 21610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:56.899583 21610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36719
I20260929 18:39:56.899617 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36719 every 8 connection(s)
I20260929 18:39:56.900502 21610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:56.903270 21678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:56.904382 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21610
I20260929 18:39:56.904508 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:56.904757 792 external_mini_cluster.cc:1428] Setting key b65e426ba83a613c66983edaf7c41712
I20260929 18:39:56.909758 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9: Bootstrap starting.
I20260929 18:39:56.910321 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:56.906257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50109 (local address 127.0.0.1:36719)
0929 18:39:56.906710 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:56.906729 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:56.906771 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:39:56.907566 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:56.907589 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:56.907953 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:56.908331 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:56.908945 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:56.908950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:56.908964 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:56.908973 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:56.909029 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:56.909053 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:56.909195 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:56.909260 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:39:56.909402 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
I20260929 18:39:56.912258 21678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:56.913079 21678 log.cc:824] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:56.913277 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:56.915155 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9: No bootstrap required, opened a new log
I20260929 18:39:56.917016 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:56.915415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58743 (local address 127.0.0.1:36719)
0929 18:39:56.915510 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:56.915515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:56.915720 (+ 205us) server_negotiation.cc:407] Connection header received
0929 18:39:56.915824 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:56.915827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:56.915859 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:56.915920 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:56.916681 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:56.916684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:56.916687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:56.916698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:56.916724 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:56.916734 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:56.916802 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:56.916874 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:39:56.916909 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:39:56.918972 21678 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcd1489837d41dca93795b3c357c7c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260929 18:39:56.919200 21678 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:56.919255 21678 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcd1489837d41dca93795b3c357c7c9, State: Initialized, Role: FOLLOWER
I20260929 18:39:56.919704 21678 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcd1489837d41dca93795b3c357c7c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260929 18:39:56.919860 21678 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:56.919930 21678 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:56.920007 21678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:56.920832 21678 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcd1489837d41dca93795b3c357c7c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260929 18:39:56.921124 21678 leader_election.cc:302] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdcd1489837d41dca93795b3c357c7c9; no voters:
I20260929 18:39:56.921377 21678 leader_election.cc:288] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:56.921475 21688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:56.921680 21688 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [term 1 LEADER]: Becoming Leader. State: Replica: fdcd1489837d41dca93795b3c357c7c9, State: Running, Role: LEADER
I20260929 18:39:56.922083 21688 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcd1489837d41dca93795b3c357c7c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260929 18:39:56.922547 21678 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:56.923682 21690 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcd1489837d41dca93795b3c357c7c9. Latest consensus state: current_term: 1 leader_uuid: "fdcd1489837d41dca93795b3c357c7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcd1489837d41dca93795b3c357c7c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } } }
I20260929 18:39:56.923811 21690 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:56.924183 21697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:56.924412 21689 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdcd1489837d41dca93795b3c357c7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcd1489837d41dca93795b3c357c7c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } } }
I20260929 18:39:56.924495 21689 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:56.927304 21697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:56.932621 21697 catalog_manager.cc:1347] Generated new cluster ID: 6a7b3fba37e94bb2a226259fd46aa890
I20260929 18:39:56.932693 21697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:56.946528 21697 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:56.946835 21697 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:56.952786 21697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdcd1489837d41dca93795b3c357c7c9: Generated new TSK 0
I20260929 18:39:56.953528 21697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:56.958815 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21610
2026-09-29T18:39:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:57 dist-test-slave-q5qx krb5kdc[21709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:57 dist-test-slave-q5qx krb5kdc[21709](info): set up 2 sockets
Sep 29 18:39:57 dist-test-slave-q5qx krb5kdc[21709](info): commencing operation
krb5kdc: starting...
W20260929 18:39:57.910190 792 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:57 dist-test-slave-q5qx krb5kdc[21709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:58.029812 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:58.138347 21722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:58.138643 21722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:58.138685 21722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:58.142808 21722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:58.142889 21722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:58.142907 21722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:58.142923 21722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:58.142936 21722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:58.148440 21722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:58.149523 21722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:58.150616 21722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:58.158916 21730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:58.158926 21727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:58.158926 21728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:58.159063 21722 server_base.cc:1034] running on GCE node
I20260929 18:39:58.159789 21722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:58.160897 21722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:58.162035 21722 hybrid_clock.cc:648] HybridClock initialized: now 1790707198162022 us; error 30 us; skew 500 ppm
Sep 29 18:39:58 dist-test-slave-q5qx krb5kdc[21709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:58.164736 21722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:58.165560 21722 webserver.cc:460] Webserver started at http://127.0.0.1:45109/ using document root <none> and password file <none>
I20260929 18:39:58.166107 21722 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:58.166183 21722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:58.166381 21722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:58.168457 21722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "274601332d1e4063abc2a17f19f3d477"
format_stamp: "Formatted at 2026-09-29 18:39:58 on dist-test-slave-q5qx"
server_key: "a217cba336e8bda1df6f97c336dadff2"
server_key_iv: "818ade9da8f54aa54ae4d7daf5405556"
server_key_version: "encryptionkey@0"
I20260929 18:39:58.168980 21722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "274601332d1e4063abc2a17f19f3d477"
format_stamp: "Formatted at 2026-09-29 18:39:58 on dist-test-slave-q5qx"
server_key: "a217cba336e8bda1df6f97c336dadff2"
server_key_iv: "818ade9da8f54aa54ae4d7daf5405556"
server_key_version: "encryptionkey@0"
I20260929 18:39:58.172637 21722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:39:58.174935 21737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:58.176162 21722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:58.176270 21722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "274601332d1e4063abc2a17f19f3d477"
format_stamp: "Formatted at 2026-09-29 18:39:58 on dist-test-slave-q5qx"
server_key: "a217cba336e8bda1df6f97c336dadff2"
server_key_iv: "818ade9da8f54aa54ae4d7daf5405556"
server_key_version: "encryptionkey@0"
I20260929 18:39:58.176364 21722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:58.188297 21722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:58.191046 21722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:58.191220 21722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:58.197913 21722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37261
I20260929 18:39:58.197954 21789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37261 every 8 connection(s)
I20260929 18:39:58.198823 21722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:58.201491 21790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:58.205843 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21722
I20260929 18:39:58.206037 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:58.206307 792 external_mini_cluster.cc:1428] Setting key 883de1891cc2978bf545bde91cf0f5d8
I20260929 18:39:58.207238 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477: Bootstrap starting.
I20260929 18:39:58.209580 21790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:58.210314 21790 log.cc:824] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:58.212364 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477: No bootstrap required, opened a new log
Sep 29 18:39:58 dist-test-slave-q5qx krb5kdc[21709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:39:58.216368 21790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274601332d1e4063abc2a17f19f3d477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260929 18:39:58.216634 21790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:58.216694 21790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274601332d1e4063abc2a17f19f3d477, State: Initialized, Role: FOLLOWER
I20260929 18:39:58.217159 21790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274601332d1e4063abc2a17f19f3d477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260929 18:39:58.217290 21790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:58.217350 21790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:58.217438 21790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:58.218334 21790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274601332d1e4063abc2a17f19f3d477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260929 18:39:58.218644 21790 leader_election.cc:302] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274601332d1e4063abc2a17f19f3d477; no voters:
I20260929 18:39:58.218925 21790 leader_election.cc:288] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:58.219022 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:58.219233 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [term 1 LEADER]: Becoming Leader. State: Replica: 274601332d1e4063abc2a17f19f3d477, State: Running, Role: LEADER
I20260929 18:39:58.219517 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274601332d1e4063abc2a17f19f3d477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260929 18:39:58.220120 21790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:58.220566 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.207734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:44683 (local address 127.0.0.1:37261)
0929 18:39:58.208193 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.208213 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.208247 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:39:58.208875 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.208892 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.209258 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.209551 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.210359 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.211539 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.212415 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.212920 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.213057 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.215744 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.215749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.215761 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.215787 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.217506 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.218006 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.218012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.218017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.218134 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.218450 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.218453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.218455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.218867 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.219063 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.219433 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.219647 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":145,"threads_started":1}
I20260929 18:39:58.220685 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274601332d1e4063abc2a17f19f3d477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274601332d1e4063abc2a17f19f3d477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260929 18:39:58.220783 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:58.220964 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274601332d1e4063abc2a17f19f3d477. Latest consensus state: current_term: 1 leader_uuid: "274601332d1e4063abc2a17f19f3d477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274601332d1e4063abc2a17f19f3d477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260929 18:39:58.221048 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:58.224239 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:39:58.225026 21810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:39:58.225102 21810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:39:58.225207 21802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:58.226115 21802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:58.232115 21802 catalog_manager.cc:1347] Generated new cluster ID: 65217a4e830a424cb135706d66260e3d
I20260929 18:39:58.232187 21802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:58.235481 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.227282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:50859 (local address 127.0.0.1:37261)
0929 18:39:58.227377 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.227382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.227396 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:39:58.227435 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.227437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.227481 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.227598 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.228303 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.229109 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.230241 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.230563 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.230676 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.232526 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.232530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.232532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.232555 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.233831 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.234303 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.234306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.234308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.234380 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.234677 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.234680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.234682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.234847 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.234975 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.235232 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.235325 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 18:39:58.270857 21802 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:58.272138 21802 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:58.284330 21802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 274601332d1e4063abc2a17f19f3d477: Generated new TSK 0
I20260929 18:39:58.285035 21802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:58.291164 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21722
W20260929 18:39:58.295895 21707 connection.cc:535] client connection to 127.0.0.1:37261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37261): Connection reset by peer (error 104)
2026-09-29T18:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:58 dist-test-slave-q5qx krb5kdc[21821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:58 dist-test-slave-q5qx krb5kdc[21821](info): set up 2 sockets
Sep 29 18:39:58 dist-test-slave-q5qx krb5kdc[21821](info): commencing operation
krb5kdc: starting...
W20260929 18:39:59.242069 792 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:39:59 dist-test-slave-q5qx krb5kdc[21821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:39:59.360735 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:59.476652 21834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:59.476933 21834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:59.476994 21834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:59.481057 21834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:59.481129 21834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:59.481155 21834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:59.481176 21834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:59.481196 21834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:59.486630 21834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:39:59.487681 21834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:59.488685 21834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:59.497092 21840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:59.497092 21842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:59.497175 21839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:59.497409 21834 server_base.cc:1034] running on GCE node
I20260929 18:39:59.497924 21834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:59.499048 21834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:59.500211 21834 hybrid_clock.cc:648] HybridClock initialized: now 1790707199500197 us; error 25 us; skew 500 ppm
Sep 29 18:39:59 dist-test-slave-q5qx krb5kdc[21821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:59.502874 21834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:39:59.503755 21834 webserver.cc:460] Webserver started at http://127.0.0.1:33927/ using document root <none> and password file <none>
I20260929 18:39:59.504258 21834 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:59.504315 21834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:59.504478 21834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:59.506481 21834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "d8477a51084a49638fb4b5103cee6180"
format_stamp: "Formatted at 2026-09-29 18:39:59 on dist-test-slave-q5qx"
server_key: "459329331793c14c37511f8d79fb790c"
server_key_iv: "a8326754d2e9bcf111cb0221a465bcc6"
server_key_version: "encryptionkey@0"
I20260929 18:39:59.506984 21834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "d8477a51084a49638fb4b5103cee6180"
format_stamp: "Formatted at 2026-09-29 18:39:59 on dist-test-slave-q5qx"
server_key: "459329331793c14c37511f8d79fb790c"
server_key_iv: "a8326754d2e9bcf111cb0221a465bcc6"
server_key_version: "encryptionkey@0"
I20260929 18:39:59.510535 21834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:59.512853 21849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:59.514196 21834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:59.514298 21834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "d8477a51084a49638fb4b5103cee6180"
format_stamp: "Formatted at 2026-09-29 18:39:59 on dist-test-slave-q5qx"
server_key: "459329331793c14c37511f8d79fb790c"
server_key_iv: "a8326754d2e9bcf111cb0221a465bcc6"
server_key_version: "encryptionkey@0"
I20260929 18:39:59.514397 21834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:59.524075 21834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:59.526881 21834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:59.527045 21834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:59.533712 21834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45831
I20260929 18:39:59.533735 21901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45831 every 8 connection(s)
I20260929 18:39:59.534673 21834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:59.536229 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21834
I20260929 18:39:59.536365 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:39:59.536623 792 external_mini_cluster.cc:1428] Setting key 6fb903193db9eb661d7b35a753d15326
I20260929 18:39:59.537889 21902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:59.544785 21902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180: Bootstrap starting.
I20260929 18:39:59.546967 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:59.538085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:35861 (local address 127.0.0.1:45831)
0929 18:39:59.538541 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:59.538561 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:59.538604 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:39:59.539440 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:59.539464 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:59.539923 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:59.540304 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:59.540740 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:59.542227 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:59.543187 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:59.543905 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:59.544058 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:59.544960 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:59.544966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:59.544982 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:59.545462 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:59.545539 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:59.545569 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:59.545692 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:59.545857 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:39:59.546035 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260929 18:39:59.547483 21902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:59.548346 21902 log.cc:824] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:59.550380 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:59.550447 21902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180: No bootstrap required, opened a new log
I20260929 18:39:59.554891 21902 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8477a51084a49638fb4b5103cee6180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260929 18:39:59.555140 21902 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:59.555179 21902 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8477a51084a49638fb4b5103cee6180, State: Initialized, Role: FOLLOWER
I20260929 18:39:59.555680 21902 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8477a51084a49638fb4b5103cee6180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260929 18:39:59.555830 21902 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:59.555896 21902 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:59.555994 21902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:59.556540 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:59.552513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:52723 (local address 127.0.0.1:45831)
0929 18:39:59.552631 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:59.552635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:59.552790 (+ 155us) server_negotiation.cc:407] Connection header received
0929 18:39:59.552922 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:59.552924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:59.552962 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:59.553024 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:59.553284 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:59.553779 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:59.554875 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:59.555112 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:59.555219 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:59.556119 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:59.556122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:59.556124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:39:59.556134 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:59.556159 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:39:59.556169 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:59.556238 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:59.556372 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:39:59.556408 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:39:59.557086 21902 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8477a51084a49638fb4b5103cee6180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260929 18:39:59.557438 21902 leader_election.cc:302] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8477a51084a49638fb4b5103cee6180; no voters:
I20260929 18:39:59.557773 21902 leader_election.cc:288] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:59.557899 21912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:59.558115 21912 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [term 1 LEADER]: Becoming Leader. State: Replica: d8477a51084a49638fb4b5103cee6180, State: Running, Role: LEADER
I20260929 18:39:59.558432 21912 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8477a51084a49638fb4b5103cee6180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } }
I20260929 18:39:59.559007 21902 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:59.560097 21913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8477a51084a49638fb4b5103cee6180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8477a51084a49638fb4b5103cee6180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } } }
I20260929 18:39:59.560247 21913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:59.560094 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8477a51084a49638fb4b5103cee6180. Latest consensus state: current_term: 1 leader_uuid: "d8477a51084a49638fb4b5103cee6180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8477a51084a49638fb4b5103cee6180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45831 } } }
I20260929 18:39:59.560364 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:59.560621 21921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:59.563704 21921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:59.569006 21921 catalog_manager.cc:1347] Generated new cluster ID: 8dd25ce978524fb898909598b879c0a4
I20260929 18:39:59.569084 21921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:59.580753 21921 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:59.581775 21921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:59.587203 21921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8477a51084a49638fb4b5103cee6180: Generated new TSK 0
I20260929 18:39:59.587899 21921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:59.597929 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21834
W20260929 18:39:59.602869 21819 connection.cc:535] client connection to 127.0.0.1:45831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45831): Connection reset by peer (error 104)
2026-09-29T18:39:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:59 dist-test-slave-q5qx krb5kdc[21933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:59 dist-test-slave-q5qx krb5kdc[21933](info): set up 2 sockets
Sep 29 18:39:59 dist-test-slave-q5qx krb5kdc[21933](info): commencing operation
krb5kdc: starting...
W20260929 18:40:00.561947 792 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:00 dist-test-slave-q5qx krb5kdc[21933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:40:00.680786 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:00.788643 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:00.788903 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:00.788944 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:00.793133 21946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:00.793203 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:00.793221 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:00.793237 21946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:00.793251 21946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:00.798738 21946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:00.799778 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:00.800793 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:40:00.809301 21946 server_base.cc:1034] running on GCE node
W20260929 18:40:00.809264 21954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:00.809264 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:00.809273 21952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:00.810005 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:00.811275 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:00.812455 21946 hybrid_clock.cc:648] HybridClock initialized: now 1790707200812445 us; error 25 us; skew 500 ppm
Sep 29 18:40:00 dist-test-slave-q5qx krb5kdc[21933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:00.815249 21946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:40:00.816138 21946 webserver.cc:460] Webserver started at http://127.0.0.1:37341/ using document root <none> and password file <none>
I20260929 18:40:00.816674 21946 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:00.816741 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:00.816929 21946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:00.819022 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "030069631a2b41ae8a74122e46fa8953"
format_stamp: "Formatted at 2026-09-29 18:40:00 on dist-test-slave-q5qx"
server_key: "ab2f082c973c85c61033c78d3b4553e8"
server_key_iv: "97bbb27349d0462616f15eec7bdf1a48"
server_key_version: "encryptionkey@0"
I20260929 18:40:00.819559 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "030069631a2b41ae8a74122e46fa8953"
format_stamp: "Formatted at 2026-09-29 18:40:00 on dist-test-slave-q5qx"
server_key: "ab2f082c973c85c61033c78d3b4553e8"
server_key_iv: "97bbb27349d0462616f15eec7bdf1a48"
server_key_version: "encryptionkey@0"
I20260929 18:40:00.823319 21946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:40:00.825771 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:00.827095 21946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:00.827221 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "030069631a2b41ae8a74122e46fa8953"
format_stamp: "Formatted at 2026-09-29 18:40:00 on dist-test-slave-q5qx"
server_key: "ab2f082c973c85c61033c78d3b4553e8"
server_key_iv: "97bbb27349d0462616f15eec7bdf1a48"
server_key_version: "encryptionkey@0"
I20260929 18:40:00.827324 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:00.831770 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:00.834815 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:00.834978 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:00.841688 21946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35815
I20260929 18:40:00.841708 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35815 every 8 connection(s)
I20260929 18:40:00.842633 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:00.845451 22014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:00.846650 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 21946
I20260929 18:40:00.846800 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:00.847093 792 external_mini_cluster.cc:1428] Setting key 81052206bd16afec3a19eda7116f79c2
I20260929 18:40:00.851871 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953: Bootstrap starting.
I20260929 18:40:00.852280 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:00.848566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:55543 (local address 127.0.0.1:35815)
0929 18:40:00.848978 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:00.848993 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:00.849030 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:40:00.849658 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:00.849673 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:00.849960 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:00.850281 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:00.851074 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:00.851078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:00.851091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:00.851102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:00.851166 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:00.851186 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:00.851293 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:00.851388 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 18:40:00.851544 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260929 18:40:00.854133 22014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:00.854844 22014 log.cc:824] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:00.854902 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:00.857106 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953: No bootstrap required, opened a new log
I20260929 18:40:00.858732 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:00.856969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:42347 (local address 127.0.0.1:35815)
0929 18:40:00.857095 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:00.857100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:00.857399 (+ 299us) server_negotiation.cc:407] Connection header received
0929 18:40:00.857486 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:00.857489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:00.857525 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:00.857607 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:00.858392 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:00.858396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:00.858399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:00.858412 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:00.858440 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:00.858454 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:00.858545 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:00.858577 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 18:40:00.858615 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:40:00.860924 22014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030069631a2b41ae8a74122e46fa8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260929 18:40:00.861157 22014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:00.861191 22014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030069631a2b41ae8a74122e46fa8953, State: Initialized, Role: FOLLOWER
I20260929 18:40:00.861584 22014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030069631a2b41ae8a74122e46fa8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260929 18:40:00.861706 22014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:00.861753 22014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:00.861824 22014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:00.862737 22014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030069631a2b41ae8a74122e46fa8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260929 18:40:00.863063 22014 leader_election.cc:302] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 030069631a2b41ae8a74122e46fa8953; no voters:
I20260929 18:40:00.863359 22014 leader_election.cc:288] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:00.863442 22024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:00.863679 22024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [term 1 LEADER]: Becoming Leader. State: Replica: 030069631a2b41ae8a74122e46fa8953, State: Running, Role: LEADER
I20260929 18:40:00.864022 22024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030069631a2b41ae8a74122e46fa8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260929 18:40:00.864485 22014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:00.865608 22025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "030069631a2b41ae8a74122e46fa8953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030069631a2b41ae8a74122e46fa8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } } }
I20260929 18:40:00.865717 22025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:00.866022 22026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 030069631a2b41ae8a74122e46fa8953. Latest consensus state: current_term: 1 leader_uuid: "030069631a2b41ae8a74122e46fa8953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030069631a2b41ae8a74122e46fa8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } } }
I20260929 18:40:00.866098 22026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:00.866106 22034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:00.868881 22034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:00.874325 22034 catalog_manager.cc:1347] Generated new cluster ID: b7d7e1b40f5443048f643bcf7e2d49be
I20260929 18:40:00.874392 22034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:00.906104 22034 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:00.906477 22034 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:00.912109 22034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 030069631a2b41ae8a74122e46fa8953: Generated new TSK 0
I20260929 18:40:00.912819 22034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:00.920953 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 21946
2026-09-29T18:40:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:00 dist-test-slave-q5qx krb5kdc[22045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:00 dist-test-slave-q5qx krb5kdc[22045](info): set up 2 sockets
Sep 29 18:40:00 dist-test-slave-q5qx krb5kdc[22045](info): commencing operation
krb5kdc: starting...
W20260929 18:40:01.878105 792 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:01 dist-test-slave-q5qx krb5kdc[22045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:40:01.998821 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:02.107717 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:02.108026 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:02.108084 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:02.112254 22058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:02.112342 22058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:02.112370 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:02.112391 22058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:02.112409 22058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:02.118147 22058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:02.119294 22058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:02.120349 22058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:02.129637 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.129712 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.129789 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:02.129925 22058 server_base.cc:1034] running on GCE node
I20260929 18:40:02.130702 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:02.131969 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:02.133188 22058 hybrid_clock.cc:648] HybridClock initialized: now 1790707202133172 us; error 34 us; skew 500 ppm
Sep 29 18:40:02 dist-test-slave-q5qx krb5kdc[22045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:02.136188 22058 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:40:02.137182 22058 webserver.cc:460] Webserver started at http://127.0.0.1:45349/ using document root <none> and password file <none>
I20260929 18:40:02.137758 22058 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:02.137828 22058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:02.138060 22058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:02.140175 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "22f1f04e7dfd4cf197db906d17e208ac"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-q5qx"
server_key: "4584f47ff0b40475cbe2f2b0855cfcfc"
server_key_iv: "3a67b3177c5a6d98d31278e282336350"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.140729 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "22f1f04e7dfd4cf197db906d17e208ac"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-q5qx"
server_key: "4584f47ff0b40475cbe2f2b0855cfcfc"
server_key_iv: "3a67b3177c5a6d98d31278e282336350"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.144754 22058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:40:02.147298 22073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.148762 22058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:40:02.148896 22058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "22f1f04e7dfd4cf197db906d17e208ac"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-q5qx"
server_key: "4584f47ff0b40475cbe2f2b0855cfcfc"
server_key_iv: "3a67b3177c5a6d98d31278e282336350"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.149008 22058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:02.168134 22058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:02.171393 22058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:02.171581 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:02.178514 22058 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45071
I20260929 18:40:02.178587 22125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45071 every 8 connection(s)
I20260929 18:40:02.179497 22058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:02.182466 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:02.185401 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22058
I20260929 18:40:02.185565 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:02.185837 792 external_mini_cluster.cc:1428] Setting key 6faede55da9e2e5fe1c8d89aaf76d6d6
I20260929 18:40:02.188768 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac: Bootstrap starting.
I20260929 18:40:02.191103 22126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:02.191845 22126 log.cc:824] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:02.193774 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac: No bootstrap required, opened a new log
Sep 29 18:40:02 dist-test-slave-q5qx krb5kdc[22045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:40:02.197029 22126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f1f04e7dfd4cf197db906d17e208ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 18:40:02.197274 22126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:02.197312 22126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f1f04e7dfd4cf197db906d17e208ac, State: Initialized, Role: FOLLOWER
I20260929 18:40:02.197744 22126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f1f04e7dfd4cf197db906d17e208ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 18:40:02.197863 22126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:02.197923 22126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:02.198056 22126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:02.198926 22126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f1f04e7dfd4cf197db906d17e208ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 18:40:02.199270 22126 leader_election.cc:302] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f1f04e7dfd4cf197db906d17e208ac; no voters:
I20260929 18:40:02.199566 22126 leader_election.cc:288] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:02.199676 22131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:02.199916 22131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [term 1 LEADER]: Becoming Leader. State: Replica: 22f1f04e7dfd4cf197db906d17e208ac, State: Running, Role: LEADER
I20260929 18:40:02.200321 22131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f1f04e7dfd4cf197db906d17e208ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 18:40:02.201077 22126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:02.201767 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:02.187455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:53777 (local address 127.0.0.1:45071)
0929 18:40:02.188045 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:02.188061 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:02.188100 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:40:02.188762 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:02.188780 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:02.189203 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:02.189537 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:02.190379 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.191436 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.192312 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.192783 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.192927 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:02.195731 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:02.195737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:02.195752 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:02.195781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:02.198086 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.198582 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.198588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.198594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.198709 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.199012 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.199015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.199017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.199498 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:02.199701 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:02.200465 (+ 764us) server_negotiation.cc:299] Negotiation successful
0929 18:40:02.200718 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":175,"threads_started":1}
I20260929 18:40:02.201960 22132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f1f04e7dfd4cf197db906d17e208ac. Latest consensus state: current_term: 1 leader_uuid: "22f1f04e7dfd4cf197db906d17e208ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f1f04e7dfd4cf197db906d17e208ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } } }
I20260929 18:40:02.202116 22132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:02.202152 22133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f1f04e7dfd4cf197db906d17e208ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f1f04e7dfd4cf197db906d17e208ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } } }
I20260929 18:40:02.202214 22133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:02.203569 22140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:02.205464 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:02.206607 22140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:02.212891 22140 catalog_manager.cc:1347] Generated new cluster ID: bb7c09b782024fadbc50b6724021d85a
I20260929 18:40:02.213050 22140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:02.215975 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:02.207739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:40385 (local address 127.0.0.1:45071)
0929 18:40:02.207852 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:02.207856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:02.208151 (+ 295us) server_negotiation.cc:407] Connection header received
0929 18:40:02.208272 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:02.208276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:02.208329 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:02.208408 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:02.209094 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.209614 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.210537 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.210781 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.210870 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:02.212523 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:02.212526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:02.212528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:02.212552 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:02.214219 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.214785 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.214788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.214790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.214867 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.215164 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.215167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.215168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.215352 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:02.215468 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:02.215737 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 18:40:02.215824 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:40:02.231166 22140 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:02.232144 22140 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:02.241669 22140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f1f04e7dfd4cf197db906d17e208ac: Generated new TSK 0
I20260929 18:40:02.242607 22140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:02.251261 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22058
W20260929 18:40:02.255820 22043 connection.cc:535] client connection to 127.0.0.1:45071 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45071): Connection reset by peer (error 104)
2026-09-29T18:40:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:02 dist-test-slave-q5qx krb5kdc[22157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:02 dist-test-slave-q5qx krb5kdc[22157](info): set up 2 sockets
Sep 29 18:40:02 dist-test-slave-q5qx krb5kdc[22157](info): commencing operation
krb5kdc: starting...
W20260929 18:40:03.202013 792 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:03 dist-test-slave-q5qx krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:40:03.324580 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:03.436766 22170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:03.437057 22170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:03.437101 22170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:03.441485 22170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:03.441566 22170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:03.441587 22170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:03.441609 22170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:03.441632 22170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:03.447296 22170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:03.448443 22170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:03.449493 22170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:03.458238 22175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.458249 22178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.458254 22176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:03.458851 22170 server_base.cc:1034] running on GCE node
I20260929 18:40:03.459393 22170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:03.460534 22170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:03.461705 22170 hybrid_clock.cc:648] HybridClock initialized: now 1790707203461681 us; error 36 us; skew 500 ppm
Sep 29 18:40:03 dist-test-slave-q5qx krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:03.464680 22170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:40:03.465729 22170 webserver.cc:460] Webserver started at http://127.0.0.1:35379/ using document root <none> and password file <none>
I20260929 18:40:03.466347 22170 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:03.466418 22170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:03.466630 22170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:03.469347 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "268ff8612921488bb776855e532b162d"
format_stamp: "Formatted at 2026-09-29 18:40:03 on dist-test-slave-q5qx"
server_key: "3a030de9cd0cc5a53f782161cfd7bb32"
server_key_iv: "c89756c47dcd803665e472136b9ef46d"
server_key_version: "encryptionkey@0"
I20260929 18:40:03.470085 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "268ff8612921488bb776855e532b162d"
format_stamp: "Formatted at 2026-09-29 18:40:03 on dist-test-slave-q5qx"
server_key: "3a030de9cd0cc5a53f782161cfd7bb32"
server_key_iv: "c89756c47dcd803665e472136b9ef46d"
server_key_version: "encryptionkey@0"
I20260929 18:40:03.474040 22170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:40:03.476850 22185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:03.478307 22170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:40:03.478434 22170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "268ff8612921488bb776855e532b162d"
format_stamp: "Formatted at 2026-09-29 18:40:03 on dist-test-slave-q5qx"
server_key: "3a030de9cd0cc5a53f782161cfd7bb32"
server_key_iv: "c89756c47dcd803665e472136b9ef46d"
server_key_version: "encryptionkey@0"
I20260929 18:40:03.478572 22170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:03.495560 22170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:03.499197 22170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:03.499396 22170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:03.506362 22170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33705
I20260929 18:40:03.506381 22237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33705 every 8 connection(s)
I20260929 18:40:03.507346 22170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:03.510262 22238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:03.511396 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22170
I20260929 18:40:03.511539 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:03.511840 792 external_mini_cluster.cc:1428] Setting key 102927c3e726ef8f15520b4be5fd9118
I20260929 18:40:03.516551 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d: Bootstrap starting.
I20260929 18:40:03.519012 22238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:03.519755 22238 log.cc:824] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:03.521368 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.513349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:52761 (local address 127.0.0.1:33705)
0929 18:40:03.513825 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.513844 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.513881 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:40:03.514605 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.514623 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.514986 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.515309 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:03.515736 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.516740 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.517688 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.518236 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.518393 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.519371 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:03.519376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:03.519391 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:03.519830 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:03.519892 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:03.519924 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:03.520045 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.520254 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.520403 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":144,"threads_started":1}
I20260929 18:40:03.522078 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d: No bootstrap required, opened a new log
I20260929 18:40:03.524789 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:03.526269 22238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268ff8612921488bb776855e532b162d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260929 18:40:03.526538 22238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:03.526626 22238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268ff8612921488bb776855e532b162d, State: Initialized, Role: FOLLOWER
I20260929 18:40:03.527160 22238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268ff8612921488bb776855e532b162d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260929 18:40:03.527333 22238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:03.527444 22238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:03.527546 22238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:03.528724 22238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268ff8612921488bb776855e532b162d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260929 18:40:03.529070 22238 leader_election.cc:302] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268ff8612921488bb776855e532b162d; no voters:
I20260929 18:40:03.529399 22238 leader_election.cc:288] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:03.529493 22248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:03.529712 22248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [term 1 LEADER]: Becoming Leader. State: Replica: 268ff8612921488bb776855e532b162d, State: Running, Role: LEADER
I20260929 18:40:03.530189 22248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268ff8612921488bb776855e532b162d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260929 18:40:03.530855 22238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:03.531298 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.526722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:39669 (local address 127.0.0.1:33705)
0929 18:40:03.526846 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.526850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.527046 (+ 196us) server_negotiation.cc:407] Connection header received
0929 18:40:03.527136 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.527139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.527194 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.527285 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:03.527596 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.528318 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.529219 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.529552 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.529671 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.530777 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:03.530780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:03.530783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:03.530797 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:03.530826 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:03.530839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:03.530899 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.531127 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.531164 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:40:03.531792 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "268ff8612921488bb776855e532b162d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268ff8612921488bb776855e532b162d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } } }
I20260929 18:40:03.531919 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:03.532553 22257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:03.532618 22250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268ff8612921488bb776855e532b162d. Latest consensus state: current_term: 1 leader_uuid: "268ff8612921488bb776855e532b162d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268ff8612921488bb776855e532b162d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } } }
I20260929 18:40:03.532709 22250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:03.535811 22257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:03.541792 22257 catalog_manager.cc:1347] Generated new cluster ID: 00f2f11bea194697aa024fb59b0563d7
I20260929 18:40:03.541857 22257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:03.555183 22257 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:03.556174 22257 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:03.566536 22257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 268ff8612921488bb776855e532b162d: Generated new TSK 0
I20260929 18:40:03.567303 22257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:03.572358 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22170
W20260929 18:40:03.577049 22155 connection.cc:535] client connection to 127.0.0.1:33705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33705): Connection reset by peer (error 104)
2026-09-29T18:40:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:03 dist-test-slave-q5qx krb5kdc[22269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:03 dist-test-slave-q5qx krb5kdc[22269](info): set up 2 sockets
Sep 29 18:40:03 dist-test-slave-q5qx krb5kdc[22269](info): commencing operation
krb5kdc: starting...
W20260929 18:40:04.526022 792 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:04 dist-test-slave-q5qx krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:40:04.648957 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:04.758677 22282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:04.758970 22282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:04.759025 22282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:04.763062 22282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:04.763137 22282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:04.763164 22282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:04.763185 22282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:04.763204 22282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:04.768579 22282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:04.769645 22282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:04.770684 22282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:04.778910 22290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:04.778877 22288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:04.778920 22287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:04.779078 22282 server_base.cc:1034] running on GCE node
I20260929 18:40:04.779639 22282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:04.780727 22282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:04.781910 22282 hybrid_clock.cc:648] HybridClock initialized: now 1790707204781871 us; error 56 us; skew 500 ppm
Sep 29 18:40:04 dist-test-slave-q5qx krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:04.794925 22282 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:40:04.795945 22282 webserver.cc:460] Webserver started at http://127.0.0.1:34763/ using document root <none> and password file <none>
I20260929 18:40:04.796487 22282 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:04.796546 22282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:04.796726 22282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:04.798883 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "3507f40f8e0b44b1b233404fbec6db6b"
format_stamp: "Formatted at 2026-09-29 18:40:04 on dist-test-slave-q5qx"
server_key: "a78c0254706b69ebe62591a8474c3e4d"
server_key_iv: "a3abe99b3ec6221cd1b4e1f4612f632e"
server_key_version: "encryptionkey@0"
I20260929 18:40:04.799410 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "3507f40f8e0b44b1b233404fbec6db6b"
format_stamp: "Formatted at 2026-09-29 18:40:04 on dist-test-slave-q5qx"
server_key: "a78c0254706b69ebe62591a8474c3e4d"
server_key_iv: "a3abe99b3ec6221cd1b4e1f4612f632e"
server_key_version: "encryptionkey@0"
I20260929 18:40:04.803381 22282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:40:04.806248 22297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.807622 22282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:40:04.807747 22282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "3507f40f8e0b44b1b233404fbec6db6b"
format_stamp: "Formatted at 2026-09-29 18:40:04 on dist-test-slave-q5qx"
server_key: "a78c0254706b69ebe62591a8474c3e4d"
server_key_iv: "a3abe99b3ec6221cd1b4e1f4612f632e"
server_key_version: "encryptionkey@0"
I20260929 18:40:04.807849 22282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:04.812318 22282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:04.815562 22282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:04.815790 22282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:04.823064 22282 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43551
I20260929 18:40:04.823230 22349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43551 every 8 connection(s)
I20260929 18:40:04.824028 22282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:04.825210 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22282
I20260929 18:40:04.825312 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:04.825568 792 external_mini_cluster.cc:1428] Setting key 8da6287e5a4143c1cc0fbb826d661467
I20260929 18:40:04.827067 22350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:04.830973 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.827158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:33943 (local address 127.0.0.1:43551)
0929 18:40:04.827600 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.827616 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.827652 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:40:04.828297 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.828320 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.828619 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.828906 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:04.829743 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:04.829748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:04.829760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:04.829770 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:04.829821 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:04.829840 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:04.829941 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.830064 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.830215 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260929 18:40:04.833237 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b: Bootstrap starting.
I20260929 18:40:04.833925 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:04.835894 22350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:04.836746 22350 log.cc:824] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:04.837823 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.836196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:52801 (local address 127.0.0.1:43551)
0929 18:40:04.836314 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.836319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.836459 (+ 140us) server_negotiation.cc:407] Connection header received
0929 18:40:04.836571 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.836573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.836607 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.836671 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:04.837420 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:04.837424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:04.837426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:04.837438 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:04.837465 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:04.837484 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:04.837572 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.837680 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.837713 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:40:04.839051 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b: No bootstrap required, opened a new log
I20260929 18:40:04.842566 22350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507f40f8e0b44b1b233404fbec6db6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43551 } }
I20260929 18:40:04.842801 22350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:04.842857 22350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3507f40f8e0b44b1b233404fbec6db6b, State: Initialized, Role: FOLLOWER
I20260929 18:40:04.843284 22350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507f40f8e0b44b1b233404fbec6db6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43551 } }
I20260929 18:40:04.843413 22350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:04.843477 22350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:04.843571 22350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:04.844473 22350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507f40f8e0b44b1b233404fbec6db6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43551 } }
I20260929 18:40:04.844831 22350 leader_election.cc:302] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3507f40f8e0b44b1b233404fbec6db6b; no voters:
I20260929 18:40:04.845157 22350 leader_election.cc:288] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:04.845259 22360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:04.845538 22360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [term 1 LEADER]: Becoming Leader. State: Replica: 3507f40f8e0b44b1b233404fbec6db6b, State: Running, Role: LEADER
I20260929 18:40:04.846372 22360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507f40f8e0b44b1b233404fbec6db6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43551 } }
I20260929 18:40:04.846547 22350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:04.848341 22361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3507f40f8e0b44b1b233404fbec6db6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507f40f8e0b44b1b233404fbec6db6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43551 } } }
I20260929 18:40:04.848548 22361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:04.848913 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3507f40f8e0b44b1b233404fbec6db6b. Latest consensus state: current_term: 1 leader_uuid: "3507f40f8e0b44b1b233404fbec6db6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507f40f8e0b44b1b233404fbec6db6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43551 } } }
I20260929 18:40:04.849476 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b [sys.catalog]: This master's current role is: LEADER
W20260929 18:40:04.851257 22374 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:04.851326 22374 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:04.851404 22375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:04.852089 22375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:04.858105 22375 catalog_manager.cc:1347] Generated new cluster ID: 882cb66d30cb4bb3a17a86b2e5b806ef
I20260929 18:40:04.858187 22375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:04.880743 22375 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:04.881145 22375 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:04.885744 22375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3507f40f8e0b44b1b233404fbec6db6b: Generated new TSK 0
I20260929 18:40:04.886482 22375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:04.894134 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22282
2026-09-29T18:40:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:04 dist-test-slave-q5qx krb5kdc[22381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:04 dist-test-slave-q5qx krb5kdc[22381](info): set up 2 sockets
Sep 29 18:40:04 dist-test-slave-q5qx krb5kdc[22381](info): commencing operation
krb5kdc: starting...
W20260929 18:40:05.858292 792 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:05 dist-test-slave-q5qx krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:05.977582 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:45321
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:45321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:06.085855 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:06.086174 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:06.086220 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:06.090273 22394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:06.090337 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:06.090355 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:06.090371 22394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:06.090386 22394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:06.095774 22394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:45321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:45321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:06.096838 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:06.097869 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:06.106703 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:06.106703 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:06.106731 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:06.106894 22394 server_base.cc:1034] running on GCE node
I20260929 18:40:06.107610 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:06.108812 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:06.110036 22394 hybrid_clock.cc:648] HybridClock initialized: now 1790707206110008 us; error 43 us; skew 500 ppm
Sep 29 18:40:06 dist-test-slave-q5qx krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:06.113183 22394 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:06.114224 22394 webserver.cc:460] Webserver started at http://127.0.198.62:39337/ using document root <none> and password file <none>
I20260929 18:40:06.114784 22394 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:06.114845 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:06.115013 22394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:06.117069 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6"
format_stamp: "Formatted at 2026-09-29 18:40:06 on dist-test-slave-q5qx"
server_key: "20d9b9004a9981da635199a12ef8b39f"
server_key_iv: "8a5d65ef8bd132fa451a09944d5bf1d7"
server_key_version: "encryptionkey@0"
I20260929 18:40:06.117583 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6"
format_stamp: "Formatted at 2026-09-29 18:40:06 on dist-test-slave-q5qx"
server_key: "20d9b9004a9981da635199a12ef8b39f"
server_key_iv: "8a5d65ef8bd132fa451a09944d5bf1d7"
server_key_version: "encryptionkey@0"
I20260929 18:40:06.121346 22394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:40:06.123939 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:06.125241 22394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:06.125351 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6"
format_stamp: "Formatted at 2026-09-29 18:40:06 on dist-test-slave-q5qx"
server_key: "20d9b9004a9981da635199a12ef8b39f"
server_key_iv: "8a5d65ef8bd132fa451a09944d5bf1d7"
server_key_version: "encryptionkey@0"
I20260929 18:40:06.125445 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:06.138777 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:06.141857 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:06.142057 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:06.149124 22394 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:45321
I20260929 18:40:06.149147 22461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:45321 every 8 connection(s)
I20260929 18:40:06.150142 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:06.152876 22462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:06.153574 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22394
I20260929 18:40:06.153705 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:06.154027 792 external_mini_cluster.cc:1428] Setting key 0af3932a60b3abf0497bb38b04d299b5
I20260929 18:40:06.158936 22462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6: Bootstrap starting.
I20260929 18:40:06.161149 22462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:06.161875 22462 log.cc:824] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:40:06 dist-test-slave-q5qx krb5kdc[22381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:06.164004 22462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6: No bootstrap required, opened a new log
I20260929 18:40:06.167999 22462 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45321 } }
I20260929 18:40:06.168224 22462 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:06.168259 22462 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ec8fbce1f04c59ad9bb2553040ddf6, State: Initialized, Role: FOLLOWER
I20260929 18:40:06.168315 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.155520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38917 (local address 127.0.198.62:45321)
0929 18:40:06.155982 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:06.156001 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:06.156037 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:40:06.156680 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:06.156698 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:06.157037 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:06.157342 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:06.158199 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.159198 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.160063 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.160534 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.160675 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:06.163485 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:06.163491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:06.163504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:06.163531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:06.165200 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.165785 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.165790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.165795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.165906 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.166320 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.166323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.166325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.166743 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:06.166918 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:06.167276 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 18:40:06.167514 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":133,"threads_started":1}
I20260929 18:40:06.168704 22462 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45321 } }
I20260929 18:40:06.168835 22462 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:06.168900 22462 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:06.168995 22462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:06.169870 22462 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45321 } }
I20260929 18:40:06.170217 22462 leader_election.cc:302] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ec8fbce1f04c59ad9bb2553040ddf6; no voters:
I20260929 18:40:06.170557 22462 leader_election.cc:288] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:06.170636 22467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:06.170881 22467 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [term 1 LEADER]: Becoming Leader. State: Replica: a6ec8fbce1f04c59ad9bb2553040ddf6, State: Running, Role: LEADER
I20260929 18:40:06.171196 22467 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45321 } }
I20260929 18:40:06.171473 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:06.171799 22462 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:06.173647 22469 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ec8fbce1f04c59ad9bb2553040ddf6. Latest consensus state: current_term: 1 leader_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45321 } } }
I20260929 18:40:06.173770 22469 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:06.173736 22468 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ec8fbce1f04c59ad9bb2553040ddf6" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45321 } } }
I20260929 18:40:06.173830 22468 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6 [sys.catalog]: This master's current role is: LEADER
W20260929 18:40:06.176878 22487 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:06.176959 22487 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:06.177024 22486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:06.177879 22486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:06.182572 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.173695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54087 (local address 127.0.198.62:45321)
0929 18:40:06.173808 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:06.173812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:06.174555 (+ 743us) server_negotiation.cc:407] Connection header received
0929 18:40:06.174678 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:06.174680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:06.174717 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:06.174780 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:06.175555 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.176297 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.177338 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.177652 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.177760 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:06.179393 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:06.179396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:06.179399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:06.179425 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:06.180760 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.181222 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.181225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.181227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.181300 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.181606 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.181609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.181611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.181812 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:06.181945 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:06.182326 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 18:40:06.182413 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:40:06.184039 22486 catalog_manager.cc:1347] Generated new cluster ID: 0ce3d6851e4f485098a2ab399c63628f
I20260929 18:40:06.184103 22486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:06.197392 22486 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:06.199064 22486 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:06.220185 22486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ec8fbce1f04c59ad9bb2553040ddf6: Generated new TSK 0
I20260929 18:40:06.221064 22486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:06.232362 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22394
W20260929 18:40:06.237017 22379 connection.cc:535] client connection to 127.0.198.62:45321 recv error: Network error: failed to read from TLS socket (remote: 127.0.198.62:45321): Connection reset by peer (error 104)
2026-09-29T18:40:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:06 dist-test-slave-q5qx krb5kdc[22493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:06 dist-test-slave-q5qx krb5kdc[22493](info): set up 2 sockets
Sep 29 18:40:06 dist-test-slave-q5qx krb5kdc[22493](info): commencing operation
krb5kdc: starting...
W20260929 18:40:07.186065 792 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:07 dist-test-slave-q5qx krb5kdc[22493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:07.306013 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:43187
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:43187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:07.415794 22506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:07.416083 22506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:07.416143 22506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:07.420238 22506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:07.420307 22506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:07.420333 22506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:07.420356 22506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:07.420383 22506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:07.425809 22506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:43187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:43187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:07.426905 22506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:07.427930 22506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:07.436334 22511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:07.436370 22514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:07.436581 22512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:07.436373 22506 server_base.cc:1034] running on GCE node
I20260929 18:40:07.437292 22506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:07.438570 22506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:07.439740 22506 hybrid_clock.cc:648] HybridClock initialized: now 1790707207439724 us; error 35 us; skew 500 ppm
Sep 29 18:40:07 dist-test-slave-q5qx krb5kdc[22493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:07.442435 22506 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:07.443383 22506 webserver.cc:460] Webserver started at http://127.0.198.62:42715/ using document root <none> and password file <none>
I20260929 18:40:07.443881 22506 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:07.443935 22506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:07.444090 22506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:07.446074 22506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "b325602842374e97b454ed5ff8d35ba5"
format_stamp: "Formatted at 2026-09-29 18:40:07 on dist-test-slave-q5qx"
server_key: "aefcc74617fce66aca663ad8084a3981"
server_key_iv: "d5d6be4656c23230e641f58d80e0d13f"
server_key_version: "encryptionkey@0"
I20260929 18:40:07.446578 22506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "b325602842374e97b454ed5ff8d35ba5"
format_stamp: "Formatted at 2026-09-29 18:40:07 on dist-test-slave-q5qx"
server_key: "aefcc74617fce66aca663ad8084a3981"
server_key_iv: "d5d6be4656c23230e641f58d80e0d13f"
server_key_version: "encryptionkey@0"
I20260929 18:40:07.450215 22506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:40:07.452747 22521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:07.454201 22506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:40:07.454308 22506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "b325602842374e97b454ed5ff8d35ba5"
format_stamp: "Formatted at 2026-09-29 18:40:07 on dist-test-slave-q5qx"
server_key: "aefcc74617fce66aca663ad8084a3981"
server_key_iv: "d5d6be4656c23230e641f58d80e0d13f"
server_key_version: "encryptionkey@0"
I20260929 18:40:07.454422 22506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:07.465579 22506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:07.468566 22506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:07.468739 22506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:07.475955 22506 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:43187
I20260929 18:40:07.475982 22573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:43187 every 8 connection(s)
I20260929 18:40:07.476899 22506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:07.479760 22574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:07.482074 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22506
I20260929 18:40:07.482199 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:07.482504 792 external_mini_cluster.cc:1428] Setting key 84d6ed6c3dd6cc40e04c10f2226013ab
I20260929 18:40:07.486337 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5: Bootstrap starting.
I20260929 18:40:07.488554 22574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:07.489269 22574 log.cc:824] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:07.491132 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5: No bootstrap required, opened a new log
I20260929 18:40:07.491750 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:07.484144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34317 (local address 127.0.198.62:43187)
0929 18:40:07.484575 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:07.484590 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:07.484626 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:40:07.485415 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:07.485434 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:07.485785 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:07.486150 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:07.486635 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:07.487701 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:07.488640 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:07.489140 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:07.489283 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:07.490142 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:07.490147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:07.490160 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:07.490537 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:07.490591 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:07.490613 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:07.490722 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:07.490884 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:40:07.491018 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260929 18:40:07.494537 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:07.494796 22574 raft_consensus.cc:348] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b325602842374e97b454ed5ff8d35ba5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43187 } }
I20260929 18:40:07.495023 22574 raft_consensus.cc:374] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:07.495075 22574 raft_consensus.cc:729] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b325602842374e97b454ed5ff8d35ba5, State: Initialized, Role: FOLLOWER
I20260929 18:40:07.495460 22574 consensus_queue.cc:260] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b325602842374e97b454ed5ff8d35ba5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43187 } }
I20260929 18:40:07.495584 22574 raft_consensus.cc:388] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:07.495687 22574 raft_consensus.cc:482] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:07.495798 22574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:07.496835 22574 raft_consensus.cc:504] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b325602842374e97b454ed5ff8d35ba5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43187 } }
I20260929 18:40:07.497155 22574 leader_election.cc:302] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b325602842374e97b454ed5ff8d35ba5; no voters:
I20260929 18:40:07.497416 22574 leader_election.cc:288] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:07.497509 22584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:07.497668 22584 raft_consensus.cc:686] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [term 1 LEADER]: Becoming Leader. State: Replica: b325602842374e97b454ed5ff8d35ba5, State: Running, Role: LEADER
I20260929 18:40:07.498039 22584 consensus_queue.cc:237] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b325602842374e97b454ed5ff8d35ba5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43187 } }
I20260929 18:40:07.498682 22574 sys_catalog.cc:564] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:07.499249 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b325602842374e97b454ed5ff8d35ba5. Latest consensus state: current_term: 1 leader_uuid: "b325602842374e97b454ed5ff8d35ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b325602842374e97b454ed5ff8d35ba5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43187 } } }
I20260929 18:40:07.499989 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:07.500401 22594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:07.500911 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:07.496452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40329 (local address 127.0.198.62:43187)
0929 18:40:07.496551 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:07.496555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:07.496753 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:40:07.496863 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:07.496866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:07.496905 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:07.496970 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:07.497354 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:07.498067 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:07.498911 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:07.499203 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:07.499292 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:07.500300 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:07.500303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:07.500306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:07.500319 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:07.500346 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:07.500359 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:07.500430 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:07.500743 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 18:40:07.500779 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:40:07.499874 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b325602842374e97b454ed5ff8d35ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b325602842374e97b454ed5ff8d35ba5" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 43187 } } }
I20260929 18:40:07.501212 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:07.503492 22594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:07.508725 22594 catalog_manager.cc:1347] Generated new cluster ID: bc4e53f36340424cad4cd6a8fbac4230
I20260929 18:40:07.508800 22594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:07.513880 22594 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:07.514806 22594 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:07.518596 22594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b325602842374e97b454ed5ff8d35ba5: Generated new TSK 0
I20260929 18:40:07.519322 22594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:07.523175 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22506
W20260929 18:40:07.527761 22491 connection.cc:535] client connection to 127.0.198.62:43187 recv error: Network error: failed to read from TLS socket (remote: 127.0.198.62:43187): Connection reset by peer (error 104)
2026-09-29T18:40:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:07 dist-test-slave-q5qx krb5kdc[22605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:07 dist-test-slave-q5qx krb5kdc[22605](info): set up 2 sockets
Sep 29 18:40:07 dist-test-slave-q5qx krb5kdc[22605](info): commencing operation
krb5kdc: starting...
W20260929 18:40:08.478026 792 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:08 dist-test-slave-q5qx krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:08.600976 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35241
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:35241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:08.714190 22618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:08.714479 22618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:08.714550 22618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:08.718597 22618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:08.718659 22618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:08.718685 22618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:08.718706 22618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:08.718726 22618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:08.724113 22618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:35241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:35241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:08.725164 22618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:08.726188 22618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:08.734340 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:08.734340 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:08.734462 22618 server_base.cc:1034] running on GCE node
W20260929 18:40:08.734416 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:08.735208 22618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:08.736300 22618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:08.737481 22618 hybrid_clock.cc:648] HybridClock initialized: now 1790707208737465 us; error 29 us; skew 500 ppm
Sep 29 18:40:08 dist-test-slave-q5qx krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:08.740285 22618 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:08.741199 22618 webserver.cc:460] Webserver started at http://127.0.198.62:37999/ using document root <none> and password file <none>
I20260929 18:40:08.741730 22618 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:08.741789 22618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:08.742081 22618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:08.744112 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "171390e9df5d4707a5b1a54fd3de7066"
format_stamp: "Formatted at 2026-09-29 18:40:08 on dist-test-slave-q5qx"
server_key: "882717a3348aba52ab8694f81551942d"
server_key_iv: "5e82bf0caf935b364221c183b11a40b6"
server_key_version: "encryptionkey@0"
I20260929 18:40:08.744622 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "171390e9df5d4707a5b1a54fd3de7066"
format_stamp: "Formatted at 2026-09-29 18:40:08 on dist-test-slave-q5qx"
server_key: "882717a3348aba52ab8694f81551942d"
server_key_iv: "5e82bf0caf935b364221c183b11a40b6"
server_key_version: "encryptionkey@0"
I20260929 18:40:08.748484 22618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:40:08.751049 22633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:08.752456 22618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:40:08.752599 22618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "171390e9df5d4707a5b1a54fd3de7066"
format_stamp: "Formatted at 2026-09-29 18:40:08 on dist-test-slave-q5qx"
server_key: "882717a3348aba52ab8694f81551942d"
server_key_iv: "5e82bf0caf935b364221c183b11a40b6"
server_key_version: "encryptionkey@0"
I20260929 18:40:08.752709 22618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:08.757448 22618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:08.760535 22618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:08.760715 22618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:08.767374 22618 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:35241
I20260929 18:40:08.767398 22685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:35241 every 8 connection(s)
I20260929 18:40:08.768288 22618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:08.771057 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:08.776571 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066: Bootstrap starting.
I20260929 18:40:08.777266 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22618
I20260929 18:40:08.777390 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:08.777670 792 external_mini_cluster.cc:1428] Setting key a20d3d891ea0907881acbed23f7bbe07
I20260929 18:40:08.779309 22686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:08.780025 22686 log.cc:824] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:08.782255 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066: No bootstrap required, opened a new log
I20260929 18:40:08.782812 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:08.778904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57121 (local address 127.0.198.62:35241)
0929 18:40:08.779342 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:08.779357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:08.779390 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:40:08.780070 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:08.780088 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:08.780420 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:08.780792 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:08.781550 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:08.781555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:08.781567 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:08.781576 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:08.781625 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:08.781643 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:08.781728 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:08.781848 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 18:40:08.782019 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260929 18:40:08.785778 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:08.786471 22686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171390e9df5d4707a5b1a54fd3de7066" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35241 } }
I20260929 18:40:08.786777 22686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:08.786837 22686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171390e9df5d4707a5b1a54fd3de7066, State: Initialized, Role: FOLLOWER
I20260929 18:40:08.787336 22686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171390e9df5d4707a5b1a54fd3de7066" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35241 } }
I20260929 18:40:08.787519 22686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:08.787590 22686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:08.787690 22686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:08.788818 22686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171390e9df5d4707a5b1a54fd3de7066" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35241 } }
I20260929 18:40:08.789140 22686 leader_election.cc:302] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 171390e9df5d4707a5b1a54fd3de7066; no voters:
I20260929 18:40:08.789440 22686 leader_election.cc:288] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:08.789525 22696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:08.789670 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:08.788061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33963 (local address 127.0.198.62:35241)
0929 18:40:08.788184 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:08.788188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:08.788364 (+ 176us) server_negotiation.cc:407] Connection header received
0929 18:40:08.788457 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:08.788459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:08.788497 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:08.788579 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:08.789268 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:08.789272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:08.789274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:08.789286 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:08.789312 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:08.789326 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:08.789415 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:08.789519 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 18:40:08.789558 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:40:08.789999 22696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [term 1 LEADER]: Becoming Leader. State: Replica: 171390e9df5d4707a5b1a54fd3de7066, State: Running, Role: LEADER
I20260929 18:40:08.790328 22696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171390e9df5d4707a5b1a54fd3de7066" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35241 } }
I20260929 18:40:08.790864 22686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:08.791517 22698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171390e9df5d4707a5b1a54fd3de7066. Latest consensus state: current_term: 1 leader_uuid: "171390e9df5d4707a5b1a54fd3de7066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171390e9df5d4707a5b1a54fd3de7066" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35241 } } }
I20260929 18:40:08.791640 22698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:08.791854 22697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "171390e9df5d4707a5b1a54fd3de7066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171390e9df5d4707a5b1a54fd3de7066" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 35241 } } }
I20260929 18:40:08.791952 22697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066 [sys.catalog]: This master's current role is: LEADER
W20260929 18:40:08.795284 22711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:08.795357 22711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:08.795430 22705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:08.796275 22705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:08.802049 22705 catalog_manager.cc:1347] Generated new cluster ID: 24c5c2bff79d4bc9b1873c2de54e40ca
I20260929 18:40:08.802237 22705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:08.812996 22705 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:08.813318 22705 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:08.817857 22705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 171390e9df5d4707a5b1a54fd3de7066: Generated new TSK 0
I20260929 18:40:08.818658 22705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:08.829417 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22618
2026-09-29T18:40:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:08 dist-test-slave-q5qx krb5kdc[22717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:08 dist-test-slave-q5qx krb5kdc[22717](info): set up 2 sockets
Sep 29 18:40:08 dist-test-slave-q5qx krb5kdc[22717](info): commencing operation
krb5kdc: starting...
W20260929 18:40:09.786108 792 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:09 dist-test-slave-q5qx krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:09.909364 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:45013
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:45013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:10.018921 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:10.019222 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:10.019280 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:10.023372 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:10.023448 22730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:10.023473 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:10.023495 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:10.023515 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:10.029059 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:45013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:45013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:10.030176 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:10.031189 22730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:10.039503 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:10.039644 22738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:10.039539 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:10.039727 22730 server_base.cc:1034] running on GCE node
I20260929 18:40:10.040374 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:10.041514 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:10.042735 22730 hybrid_clock.cc:648] HybridClock initialized: now 1790707210042708 us; error 40 us; skew 500 ppm
Sep 29 18:40:10 dist-test-slave-q5qx krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:10.052682 22730 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:10.053701 22730 webserver.cc:460] Webserver started at http://127.0.198.62:43747/ using document root <none> and password file <none>
I20260929 18:40:10.054311 22730 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:10.054376 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:10.054544 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:10.056674 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "16960ef0c9da408cb935bd481dc4662c"
format_stamp: "Formatted at 2026-09-29 18:40:10 on dist-test-slave-q5qx"
server_key: "ac76b4a0a8e473d74ddf6226e0da8a75"
server_key_iv: "1ca350d2710367bb0686a967b85bae06"
server_key_version: "encryptionkey@0"
I20260929 18:40:10.057211 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "16960ef0c9da408cb935bd481dc4662c"
format_stamp: "Formatted at 2026-09-29 18:40:10 on dist-test-slave-q5qx"
server_key: "ac76b4a0a8e473d74ddf6226e0da8a75"
server_key_iv: "1ca350d2710367bb0686a967b85bae06"
server_key_version: "encryptionkey@0"
I20260929 18:40:10.060981 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 18:40:10.063580 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:10.064843 22730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:40:10.064957 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "16960ef0c9da408cb935bd481dc4662c"
format_stamp: "Formatted at 2026-09-29 18:40:10 on dist-test-slave-q5qx"
server_key: "ac76b4a0a8e473d74ddf6226e0da8a75"
server_key_iv: "1ca350d2710367bb0686a967b85bae06"
server_key_version: "encryptionkey@0"
I20260929 18:40:10.065093 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:10.081530 22730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:10.084637 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:10.084789 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:10.091817 22730 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:45013
I20260929 18:40:10.091836 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:45013 every 8 connection(s)
I20260929 18:40:10.092746 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:10.095592 22798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:10.095671 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22730
I20260929 18:40:10.095809 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:10.096077 792 external_mini_cluster.cc:1428] Setting key 865c9e8a82ce59fd67f5480ccaf0a05f
I20260929 18:40:10.102837 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c: Bootstrap starting.
I20260929 18:40:10.105551 22798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c: Neither blocks nor log segments found. Creating new log.
Sep 29 18:40:10 dist-test-slave-q5qx krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:10.106379 22798 log.cc:824] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:10.108955 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c: No bootstrap required, opened a new log
I20260929 18:40:10.111589 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:10.097431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58169 (local address 127.0.198.62:45013)
0929 18:40:10.097864 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:10.097884 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:10.097928 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:40:10.098812 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:10.098840 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:10.099301 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:10.099681 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:10.100393 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:10.101873 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:10.102844 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:10.103587 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:10.103746 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:10.106338 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:10.106345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:10.106364 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:10.106399 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:10.108604 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:10.109177 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:10.109183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:10.109188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:10.109310 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:10.109597 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:10.109600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:10.109602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:10.110064 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:10.110256 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:10.110540 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:40:10.110837 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
I20260929 18:40:10.112798 22798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16960ef0c9da408cb935bd481dc4662c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45013 } }
I20260929 18:40:10.113029 22798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:10.113081 22798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16960ef0c9da408cb935bd481dc4662c, State: Initialized, Role: FOLLOWER
I20260929 18:40:10.113512 22798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16960ef0c9da408cb935bd481dc4662c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45013 } }
I20260929 18:40:10.113660 22798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:10.113731 22798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:10.113822 22798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:10.114580 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:10.114929 22798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16960ef0c9da408cb935bd481dc4662c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45013 } }
I20260929 18:40:10.115263 22798 leader_election.cc:302] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16960ef0c9da408cb935bd481dc4662c; no voters:
I20260929 18:40:10.115562 22798 leader_election.cc:288] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:10.115659 22803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:10.115882 22803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [term 1 LEADER]: Becoming Leader. State: Replica: 16960ef0c9da408cb935bd481dc4662c, State: Running, Role: LEADER
I20260929 18:40:10.116201 22803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16960ef0c9da408cb935bd481dc4662c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45013 } }
I20260929 18:40:10.116959 22798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:10.117421 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16960ef0c9da408cb935bd481dc4662c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16960ef0c9da408cb935bd481dc4662c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45013 } } }
I20260929 18:40:10.117538 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:10.117730 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16960ef0c9da408cb935bd481dc4662c. Latest consensus state: current_term: 1 leader_uuid: "16960ef0c9da408cb935bd481dc4662c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16960ef0c9da408cb935bd481dc4662c" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 45013 } } }
I20260929 18:40:10.117816 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:10.118059 22812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:10.121498 22812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:10.125451 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:10.116586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36899 (local address 127.0.198.62:45013)
0929 18:40:10.116699 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:10.116703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:10.116935 (+ 232us) server_negotiation.cc:407] Connection header received
0929 18:40:10.117051 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:10.117054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:10.117104 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:10.117169 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:10.117890 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:10.118703 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:10.119817 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:10.120140 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:10.120213 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:10.121956 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:10.121960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:10.121963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:10.122013 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:10.123688 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:10.124165 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:10.124169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:10.124171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:10.124260 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:10.124541 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:10.124544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:10.124545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:10.124785 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:10.124913 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:10.125162 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 18:40:10.125289 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:40:10.127851 22812 catalog_manager.cc:1347] Generated new cluster ID: b7f360b3e159424daa52c1a3e564ddea
I20260929 18:40:10.127915 22812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:10.144057 22812 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:10.145274 22812 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:10.150243 22812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16960ef0c9da408cb935bd481dc4662c: Generated new TSK 0
I20260929 18:40:10.150985 22812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:10.160131 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22730
2026-09-29T18:40:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:10 dist-test-slave-q5qx krb5kdc[22829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:10 dist-test-slave-q5qx krb5kdc[22829](info): set up 2 sockets
Sep 29 18:40:10 dist-test-slave-q5qx krb5kdc[22829](info): commencing operation
krb5kdc: starting...
W20260929 18:40:11.122149 792 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:11 dist-test-slave-q5qx krb5kdc[22829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:11.241452 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42729
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:11.348636 22842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:11.348910 22842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:11.348953 22842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:11.352893 22842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:11.352957 22842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:11.352975 22842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:11.352991 22842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:11.353005 22842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:11.358242 22842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:11.359280 22842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:11.360246 22842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:40:11.368799 22842 server_base.cc:1034] running on GCE node
W20260929 18:40:11.368736 22850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:11.368733 22848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:11.368752 22847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:11.369573 22842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:11.370687 22842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:11.371872 22842 hybrid_clock.cc:648] HybridClock initialized: now 1790707211371844 us; error 40 us; skew 500 ppm
Sep 29 18:40:11 dist-test-slave-q5qx krb5kdc[22829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:11.374657 22842 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:11.375562 22842 webserver.cc:460] Webserver started at http://127.0.198.62:42185/ using document root <none> and password file <none>
I20260929 18:40:11.376300 22842 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:11.376377 22842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:11.376591 22842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:11.378741 22842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "36022e89ca74495097f61624b598467f"
format_stamp: "Formatted at 2026-09-29 18:40:11 on dist-test-slave-q5qx"
server_key: "b8069281fad5a7cda1b3fd6e301a8647"
server_key_iv: "acbac2fb4a7b9dfdd8e5efcdd1b8778e"
server_key_version: "encryptionkey@0"
I20260929 18:40:11.379305 22842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "36022e89ca74495097f61624b598467f"
format_stamp: "Formatted at 2026-09-29 18:40:11 on dist-test-slave-q5qx"
server_key: "b8069281fad5a7cda1b3fd6e301a8647"
server_key_iv: "acbac2fb4a7b9dfdd8e5efcdd1b8778e"
server_key_version: "encryptionkey@0"
I20260929 18:40:11.383118 22842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:40:11.385571 22857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:11.386853 22842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:11.386986 22842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "36022e89ca74495097f61624b598467f"
format_stamp: "Formatted at 2026-09-29 18:40:11 on dist-test-slave-q5qx"
server_key: "b8069281fad5a7cda1b3fd6e301a8647"
server_key_iv: "acbac2fb4a7b9dfdd8e5efcdd1b8778e"
server_key_version: "encryptionkey@0"
I20260929 18:40:11.387120 22842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:11.405530 22842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:11.408749 22842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:11.408913 22842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:11.415875 22842 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:42729
I20260929 18:40:11.415892 22909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:42729 every 8 connection(s)
I20260929 18:40:11.416837 22842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:11.417765 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22842
I20260929 18:40:11.417879 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:11.418193 792 external_mini_cluster.cc:1428] Setting key 922cb8abd0ff8de78b99d7441a30ac6d
I20260929 18:40:11.419965 22910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:11.427026 22910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f: Bootstrap starting.
I20260929 18:40:11.428562 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:11.419618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40817 (local address 127.0.198.62:42729)
0929 18:40:11.420039 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:11.420059 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:11.420101 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:40:11.420920 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:11.420942 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:11.421372 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:11.421749 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:11.422176 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:11.423689 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:11.424674 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:11.425366 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:11.425525 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:11.426507 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:11.426514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:11.426531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:11.427040 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:11.427110 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:11.427143 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:11.427237 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:11.427404 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:40:11.427631 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260929 18:40:11.429747 22910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:11.430639 22910 log.cc:824] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:11.431617 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:11.432739 22910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f: No bootstrap required, opened a new log
I20260929 18:40:11.436299 22910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36022e89ca74495097f61624b598467f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42729 } }
I20260929 18:40:11.436584 22910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:11.436650 22910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36022e89ca74495097f61624b598467f, State: Initialized, Role: FOLLOWER
I20260929 18:40:11.437129 22910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36022e89ca74495097f61624b598467f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42729 } }
I20260929 18:40:11.437278 22910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:11.437397 22910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:11.437408 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:11.433616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33339 (local address 127.0.198.62:42729)
0929 18:40:11.433741 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:11.433745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:11.433940 (+ 195us) server_negotiation.cc:407] Connection header received
0929 18:40:11.434096 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:11.434099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:11.434145 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:11.434207 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:11.434465 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:11.434996 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:11.435761 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:11.436002 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:11.436089 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:11.436891 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:11.436895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:11.436898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:11.436910 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:11.436939 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:11.436952 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:11.437011 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:11.437169 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 18:40:11.437230 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:40:11.437518 22910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:11.438584 22910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36022e89ca74495097f61624b598467f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42729 } }
I20260929 18:40:11.438894 22910 leader_election.cc:302] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36022e89ca74495097f61624b598467f; no voters:
I20260929 18:40:11.439157 22910 leader_election.cc:288] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:11.439255 22920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:11.439460 22920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [term 1 LEADER]: Becoming Leader. State: Replica: 36022e89ca74495097f61624b598467f, State: Running, Role: LEADER
I20260929 18:40:11.439846 22920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36022e89ca74495097f61624b598467f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42729 } }
I20260929 18:40:11.440304 22910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:11.441588 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36022e89ca74495097f61624b598467f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36022e89ca74495097f61624b598467f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42729 } } }
I20260929 18:40:11.441586 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36022e89ca74495097f61624b598467f. Latest consensus state: current_term: 1 leader_uuid: "36022e89ca74495097f61624b598467f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36022e89ca74495097f61624b598467f" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42729 } } }
I20260929 18:40:11.441746 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:11.441746 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:11.442162 22929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:11.445745 22929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:11.451838 22929 catalog_manager.cc:1347] Generated new cluster ID: 023e2ce8b4dd4c408c90a367922a44b3
I20260929 18:40:11.451912 22929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:11.487327 22929 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:11.488315 22929 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:11.492843 22929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36022e89ca74495097f61624b598467f: Generated new TSK 0
I20260929 18:40:11.493592 22929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:11.504326 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22842
2026-09-29T18:40:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:11 dist-test-slave-q5qx krb5kdc[22941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:11 dist-test-slave-q5qx krb5kdc[22941](info): set up 2 sockets
Sep 29 18:40:11 dist-test-slave-q5qx krb5kdc[22941](info): commencing operation
krb5kdc: starting...
W20260929 18:40:12.458256 792 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:12 dist-test-slave-q5qx krb5kdc[22941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:12.577433 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42123
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:12.687884 22954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:12.688187 22954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:12.688230 22954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:12.692456 22954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:12.692521 22954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:12.692539 22954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:12.692554 22954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:12.692569 22954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:12.698065 22954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:12.699144 22954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:12.700152 22954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:12.708489 22960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:12.708500 22962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:12.708500 22959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:12.708545 22954 server_base.cc:1034] running on GCE node
I20260929 18:40:12.709270 22954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:12.710474 22954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:12.711645 22954 hybrid_clock.cc:648] HybridClock initialized: now 1790707212711630 us; error 42 us; skew 500 ppm
Sep 29 18:40:12 dist-test-slave-q5qx krb5kdc[22941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:12.714668 22954 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:12.715610 22954 webserver.cc:460] Webserver started at http://127.0.198.62:44829/ using document root <none> and password file <none>
I20260929 18:40:12.716161 22954 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:12.716223 22954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:12.716419 22954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:12.718596 22954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "99534f7ca1114b36a82e803ea35e85e1"
format_stamp: "Formatted at 2026-09-29 18:40:12 on dist-test-slave-q5qx"
server_key: "56c10b147d0829dd8ce418a2b1cad9e0"
server_key_iv: "e60656fdfc12a11ed4d48e4b21cb8d3d"
server_key_version: "encryptionkey@0"
I20260929 18:40:12.719128 22954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "99534f7ca1114b36a82e803ea35e85e1"
format_stamp: "Formatted at 2026-09-29 18:40:12 on dist-test-slave-q5qx"
server_key: "56c10b147d0829dd8ce418a2b1cad9e0"
server_key_iv: "e60656fdfc12a11ed4d48e4b21cb8d3d"
server_key_version: "encryptionkey@0"
I20260929 18:40:12.723069 22954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:40:12.725528 22969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:12.726873 22954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:12.726986 22954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "99534f7ca1114b36a82e803ea35e85e1"
format_stamp: "Formatted at 2026-09-29 18:40:12 on dist-test-slave-q5qx"
server_key: "56c10b147d0829dd8ce418a2b1cad9e0"
server_key_iv: "e60656fdfc12a11ed4d48e4b21cb8d3d"
server_key_version: "encryptionkey@0"
I20260929 18:40:12.727108 22954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:12.731731 22954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:12.734735 22954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:12.734895 22954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:12.741461 22954 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:42123
I20260929 18:40:12.741494 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:42123 every 8 connection(s)
I20260929 18:40:12.742445 22954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:12.743533 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 22954
I20260929 18:40:12.743654 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:12.743911 792 external_mini_cluster.cc:1428] Setting key 7ceb213e572203f7a6ce32889be0f3ca
I20260929 18:40:12.745662 23022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:12.749449 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:12.745579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54875 (local address 127.0.198.62:42123)
0929 18:40:12.746064 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:12.746080 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:12.746118 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:40:12.746782 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:12.746800 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:12.747175 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:12.747503 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:12.748300 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:12.748304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:12.748317 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:12.748327 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:12.748379 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:12.748397 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:12.748526 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:12.748600 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 18:40:12.748753 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260929 18:40:12.751847 23022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1: Bootstrap starting.
I20260929 18:40:12.752534 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:12.754117 23022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:12.754871 23022 log.cc:824] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:12.756177 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:12.754510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35037 (local address 127.0.198.62:42123)
0929 18:40:12.754632 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:12.754637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:12.754847 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:40:12.754949 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:12.754952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:12.754988 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:12.755055 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:12.755802 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:12.755806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:12.755808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:40:12.755819 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:12.755847 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:40:12.755857 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:12.755924 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:12.756008 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 18:40:12.756045 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:40:12.757061 23022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1: No bootstrap required, opened a new log
I20260929 18:40:12.760636 23022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99534f7ca1114b36a82e803ea35e85e1" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42123 } }
I20260929 18:40:12.760871 23022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:12.760931 23022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99534f7ca1114b36a82e803ea35e85e1, State: Initialized, Role: FOLLOWER
I20260929 18:40:12.761360 23022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99534f7ca1114b36a82e803ea35e85e1" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42123 } }
I20260929 18:40:12.761498 23022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:12.761567 23022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:12.761668 23022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:12.762662 23022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99534f7ca1114b36a82e803ea35e85e1" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42123 } }
I20260929 18:40:12.762990 23022 leader_election.cc:302] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99534f7ca1114b36a82e803ea35e85e1; no voters:
I20260929 18:40:12.763242 23022 leader_election.cc:288] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:12.763350 23032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:12.763556 23032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [term 1 LEADER]: Becoming Leader. State: Replica: 99534f7ca1114b36a82e803ea35e85e1, State: Running, Role: LEADER
I20260929 18:40:12.763859 23032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99534f7ca1114b36a82e803ea35e85e1" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42123 } }
I20260929 18:40:12.764343 23022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:12.765467 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99534f7ca1114b36a82e803ea35e85e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99534f7ca1114b36a82e803ea35e85e1" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42123 } } }
I20260929 18:40:12.765586 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:12.765961 23041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:12.766194 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99534f7ca1114b36a82e803ea35e85e1. Latest consensus state: current_term: 1 leader_uuid: "99534f7ca1114b36a82e803ea35e85e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99534f7ca1114b36a82e803ea35e85e1" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 42123 } } }
I20260929 18:40:12.766285 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:12.769237 23041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:12.774801 23041 catalog_manager.cc:1347] Generated new cluster ID: 9e6854eb93c2460bb1165eb4696ff209
I20260929 18:40:12.774875 23041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:12.792513 23041 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:12.792835 23041 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:12.798442 23041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99534f7ca1114b36a82e803ea35e85e1: Generated new TSK 0
I20260929 18:40:12.799153 23041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:12.803560 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 22954
2026-09-29T18:40:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1299 ms)
[----------] 18 tests from AuthTokenIssuingTest (23822 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:12 dist-test-slave-q5qx krb5kdc[23053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:12 dist-test-slave-q5qx krb5kdc[23053](info): set up 2 sockets
Sep 29 18:40:12 dist-test-slave-q5qx krb5kdc[23053](info): commencing operation
krb5kdc: starting...
W20260929 18:40:13.766005 792 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:13 dist-test-slave-q5qx krb5kdc[23053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:13.890316 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:33333
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:33333,127.0.198.61:36109,127.0.198.60:42615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:14.000379 23066 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:40:14.000694 23066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:14.000739 23066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:14.000805 23066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:14.005012 23066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:14.005103 23066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:14.005132 23066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:14.005154 23066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:14.005187 23066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:14.011610 23066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:33333,127.0.198.61:36109,127.0.198.60:42615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:14.012873 23066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:14.013950 23066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:40:14.023698 23066 server_base.cc:1034] running on GCE node
W20260929 18:40:14.023620 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:14.023640 23072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:14.023663 23071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:14.024448 23066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:14.025650 23066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:14.026844 23066 hybrid_clock.cc:648] HybridClock initialized: now 1790707214026803 us; error 53 us; skew 500 ppm
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:14.029735 23066 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:14.030689 23066 webserver.cc:460] Webserver started at http://127.0.198.62:40405/ using document root <none> and password file <none>
I20260929 18:40:14.031234 23066 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:14.031289 23066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:14.031456 23066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:14.033563 23066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "e4bc4d093dc646a5992cf2b0ee8a3222"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "d1fa33e562d8245336569f8ede6a5030"
server_key_iv: "60fd6ac9d6cdbcc27ca4ddea06f64ecf"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.034140 23066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "e4bc4d093dc646a5992cf2b0ee8a3222"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "d1fa33e562d8245336569f8ede6a5030"
server_key_iv: "60fd6ac9d6cdbcc27ca4ddea06f64ecf"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.037920 23066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:40:14.040725 23081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:14.042232 23066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:40:14.042352 23066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "e4bc4d093dc646a5992cf2b0ee8a3222"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "d1fa33e562d8245336569f8ede6a5030"
server_key_iv: "60fd6ac9d6cdbcc27ca4ddea06f64ecf"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.042470 23066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:14.056440 23066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:14.060077 23066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:14.060273 23066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:14.067734 23066 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:33333
I20260929 18:40:14.067981 23133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:33333 every 8 connection(s)
I20260929 18:40:14.068809 23066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:14.071910 23134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:14.075903 23134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:14.077211 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 23066
I20260929 18:40:14.077345 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:14.077630 792 external_mini_cluster.cc:1428] Setting key fbd019cf48f20e791c7cb5a4f4407a1a
WARNING: no policy specified for kudu/127.0.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.61@KRBTEST.COM" created.
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:14.094797 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.079411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.0.198.62:33333)
0929 18:40:14.079753 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.079770 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.079818 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:40:14.080334 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.080341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.080548 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.080713 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.081775 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.082740 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.084311 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.084733 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.084866 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.089771 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.089780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.089795 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.089826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.091530 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.092106 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.092111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.092112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.092246 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.092613 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.092617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.092618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.093032 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.093162 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.093646 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.093883 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
I20260929 18:40:14.094800 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.078467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:33333 (local address 127.0.0.1:43744)
0929 18:40:14.079010 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.079075 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.080010 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.080776 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.080782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.081221 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.081602 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.081626 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.082874 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.082877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.084150 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.084163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.084909 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.084914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.085092 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.085777 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.085798 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.089577 (+ 3779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.091653 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.091662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.091674 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.091976 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.092356 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.092360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.092362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.092518 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.093162 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.093170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.093351 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.093559 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.094009 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":120,"thread_start_us":128,"threads_started":1}
I20260929 18:40:14.097486 23134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:14.098347 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.098004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39030)
0929 18:40:14.098182 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.098255 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 18:40:14.098565 23084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
I20260929 18:40:14.099504 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.099294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39034)
0929 18:40:14.099404 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.099450 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:40:14.099685 23134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:36109: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab.
Entry for principal kudu/127.0.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab.
I20260929 18:40:14.106593 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:36109
--webserver_interface=127.0.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab
--principal=kudu/127.0.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:33333,127.0.198.61:36109,127.0.198.60:42615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:14.148940 23134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } attempt: 1
I20260929 18:40:14.149861 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.149521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39046)
0929 18:40:14.149714 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.149779 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 18:40:14.150712 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.150430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39054)
0929 18:40:14.150572 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.150644 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 18:40:14.150866 23134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:36109: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
I20260929 18:40:14.219060 23134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } attempt: 2
I20260929 18:40:14.219921 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.219625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39060)
0929 18:40:14.219787 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.219844 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 18:40:14.220650 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.220419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39076)
0929 18:40:14.220538 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.220586 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 18:40:14.220785 23134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:36109: Network error: Client connection negotiation failed: client connection to 127.0.198.61:36109: connect: Connection refused (error 111)
W20260929 18:40:14.223919 23141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:40:14.224229 23141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:14.224280 23141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:14.224347 23141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:14.228901 23141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:14.228995 23141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:14.229027 23141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:14.229048 23141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:14.229069 23141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:14.234905 23141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:33333,127.0.198.61:36109,127.0.198.60:42615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab
--principal=kudu/127.0.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:14.236259 23141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:14.237375 23141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:14.246320 23147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:14.246285 23149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:14.246299 23146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:14.246353 23141 server_base.cc:1034] running on GCE node
I20260929 18:40:14.247108 23141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:14.248302 23141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:14.249509 23141 hybrid_clock.cc:648] HybridClock initialized: now 1790707214249495 us; error 34 us; skew 500 ppm
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:14.252629 23141 init.cc:377] Logged in from keytab as kudu/127.0.198.61@KRBTEST.COM (short username kudu)
I20260929 18:40:14.253695 23141 webserver.cc:460] Webserver started at http://127.0.198.61:37485/ using document root <none> and password file <none>
I20260929 18:40:14.254313 23141 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:14.254385 23141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:14.254601 23141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:14.256750 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data/instance:
uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "27c9d2e3ac0768c29103b7fb3dcdb030"
server_key_iv: "8041f71e3511c5024034b41b7088050c"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.257321 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/wal/instance:
uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "27c9d2e3ac0768c29103b7fb3dcdb030"
server_key_iv: "8041f71e3511c5024034b41b7088050c"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.261396 23141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:40:14.264228 23156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:14.265673 23141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 18:40:14.265803 23141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/wal
uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "27c9d2e3ac0768c29103b7fb3dcdb030"
server_key_iv: "8041f71e3511c5024034b41b7088050c"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.265916 23141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:14.299402 23141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:14.302784 23141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:14.302999 23141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:14.310570 23141 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.61:36109
I20260929 18:40:14.310842 23208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.61:36109 every 8 connection(s)
I20260929 18:40:14.311651 23141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
I20260929 18:40:14.311931 23134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } attempt: 3
I20260929 18:40:14.313755 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 23141
I20260929 18:40:14.313870 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-1/wal/instance
I20260929 18:40:14.314183 792 external_mini_cluster.cc:1428] Setting key 0de3f8c9862d42e8bb299dd117e79a1a
I20260929 18:40:14.315285 23209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:14.320379 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.198.60@KRBTEST.COM; defaulting to no policy
I20260929 18:40:14.326985 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.312437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39090)
0929 18:40:14.312599 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.312616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.312723 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.315520 (+ 2797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.315524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.315727 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.315881 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.315890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.318218 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.318222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.319048 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.319063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.320215 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.320219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.320339 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.320967 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.320985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.322459 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.324931 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.324934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.324937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.325142 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.325523 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.325526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.325528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.325623 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.326421 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.326423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.326520 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.326641 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.326790 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.60@KRBTEST.COM" created.
I20260929 18:40:14.328171 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.313126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.198.61:36109)
0929 18:40:14.313593 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.313613 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.313659 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:40:14.314644 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.314668 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.315140 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.316122 (+ 982us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.316562 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.318050 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.319210 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.320043 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.320228 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.322713 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.322720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.322738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.322775 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.324825 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.325255 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.325261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.325267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.325416 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.325712 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.325715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.325717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.326272 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.326407 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.326778 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.327080 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:14.330327 23134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:14.331091 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.330803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49102)
0929 18:40:14.330956 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.331014 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:40:14.332209 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.331896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49112)
0929 18:40:14.332086 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.332138 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 18:40:14.332371 23134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:42615: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
I20260929 18:40:14.334282 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.322745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.0.198.62:33333)
0929 18:40:14.322864 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.322869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.323581 (+ 712us) server_negotiation.cc:407] Connection header received
0929 18:40:14.323757 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.323760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.323819 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.323911 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.324813 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.325402 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.326611 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.327482 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.327611 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.330270 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.330274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.330277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.330301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.332066 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.332626 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.332628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.332630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.332706 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.333185 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.333192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.333193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.333448 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.333539 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.334059 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.334171 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:40:14.334364 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.322735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:33333 (local address 127.0.0.1:43754)
0929 18:40:14.323456 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.323516 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.323642 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.323952 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.323961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.324425 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.324601 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.324609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.325494 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.325497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.326474 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.326482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.327612 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.327616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.327741 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.328407 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.328420 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.330122 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.332195 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.332207 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.332213 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.332526 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.332797 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.332800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.332802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.332901 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.333580 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.333587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.333748 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.333916 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.334166 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":157,"threads_started":1}
I20260929 18:40:14.335538 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
I20260929 18:40:14.344359 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.336119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39094)
0929 18:40:14.336279 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.336293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.336355 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.336592 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.336594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.336730 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.336830 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.336835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.337724 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.337727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.338388 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.338397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.338919 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.338932 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.339024 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.339524 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.339536 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.340948 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.342796 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.342799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.342801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.343023 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.343296 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.343299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.343300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.343391 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.343841 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.343843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.343929 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.344088 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.344226 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 18:40:14.344491 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.336213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.0.198.61:36109)
0929 18:40:14.336331 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.336335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.336351 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:40:14.336417 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.336420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.336483 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.336565 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.336965 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.337602 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.338517 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.338778 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.338892 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.341141 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.341145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.341148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.341174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.342678 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.343109 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.343112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.343113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.343185 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.343508 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.343510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.343511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.343733 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.343823 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.344152 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.344303 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":43}
I20260929 18:40:14.345278 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:14.346251 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.345692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49116)
0929 18:40:14.345896 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.346158 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260929 18:40:14.346525 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
I20260929 18:40:14.347539 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.347334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49122)
0929 18:40:14.347432 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.347483 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:40:14.347661 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:42615: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab.
Entry for principal kudu/127.0.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab.
I20260929 18:40:14.349383 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:42615
--webserver_interface=127.0.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab
--principal=kudu/127.0.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:33333,127.0.198.61:36109,127.0.198.60:42615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:14.363534 23134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } attempt: 1
I20260929 18:40:14.364423 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.364094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49130)
0929 18:40:14.364257 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.364331 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 18:40:14.365226 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.364999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49136)
0929 18:40:14.365103 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.365155 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:40:14.365429 23134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:42615: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
I20260929 18:40:14.396981 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } attempt: 1
I20260929 18:40:14.398010 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.397611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49148)
0929 18:40:14.397796 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.397875 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 18:40:14.398733 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.398503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49160)
0929 18:40:14.398616 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.398669 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:40:14.398977 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:42615: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
I20260929 18:40:14.440642 23134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } attempt: 2
I20260929 18:40:14.441529 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.441190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49166)
0929 18:40:14.441382 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.441447 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 18:40:14.442509 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.442190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49172)
0929 18:40:14.442346 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.442392 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 18:40:14.442682 23134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:42615: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
W20260929 18:40:14.465867 23216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:40:14.466217 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:14.466243 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:14.466280 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:40:14.467155 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } attempt: 2
I20260929 18:40:14.467991 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.467656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49186)
0929 18:40:14.467825 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.467899 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:40:14.468808 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.468589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49190)
0929 18:40:14.468694 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.468745 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:40:14.468998 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:42615: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
W20260929 18:40:14.470872 23216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:14.470927 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:14.470945 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:14.470964 23216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:14.470992 23216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:14.476573 23216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:33333,127.0.198.61:36109,127.0.198.60:42615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab
--principal=kudu/127.0.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:42615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:14.477661 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:14.478749 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:14.487149 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:14.487278 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:14.487154 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:14.487368 23216 server_base.cc:1034] running on GCE node
I20260929 18:40:14.488116 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:14.489348 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:14.490527 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790707214490507 us; error 39 us; skew 500 ppm
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:14.493342 23216 init.cc:377] Logged in from keytab as kudu/127.0.198.60@KRBTEST.COM (short username kudu)
I20260929 18:40:14.494388 23216 webserver.cc:460] Webserver started at http://127.0.198.60:41247/ using document root <none> and password file <none>
I20260929 18:40:14.494956 23216 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:14.495069 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:14.495280 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:14.497352 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data/instance:
uuid: "7d9c217bac944432aadf1b839992a6dd"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "8ff0cebb3f4156db8e3a3b2de77c62a9"
server_key_iv: "3ca7d5d97f4534346b4b9d953d64d780"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.497918 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/wal/instance:
uuid: "7d9c217bac944432aadf1b839992a6dd"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "8ff0cebb3f4156db8e3a3b2de77c62a9"
server_key_iv: "3ca7d5d97f4534346b4b9d953d64d780"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.501731 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:40:14.504282 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:14.505949 23216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:14.506130 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/wal
uuid: "7d9c217bac944432aadf1b839992a6dd"
format_stamp: "Formatted at 2026-09-29 18:40:14 on dist-test-slave-q5qx"
server_key: "8ff0cebb3f4156db8e3a3b2de77c62a9"
server_key_iv: "3ca7d5d97f4534346b4b9d953d64d780"
server_key_version: "encryptionkey@0"
I20260929 18:40:14.506263 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:14.532832 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:14.536375 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:14.536566 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:14.541843 23134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } attempt: 3
I20260929 18:40:14.542713 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.542394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49196)
0929 18:40:14.542514 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.542622 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:40:14.543427 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.543219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49204)
0929 18:40:14.543323 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.543371 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 18:40:14.543570 23134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:42615: Network error: Client connection negotiation failed: client connection to 127.0.198.60:42615: connect: Connection refused (error 111)
I20260929 18:40:14.544071 23216 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.60:42615
I20260929 18:40:14.544309 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.60:42615 every 8 connection(s)
I20260929 18:40:14.545090 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
I20260929 18:40:14.546101 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 23216
I20260929 18:40:14.546218 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706942746699-792-0/minicluster-data/master-2/wal/instance
I20260929 18:40:14.546510 792 external_mini_cluster.cc:1428] Setting key a5dae491156b7cf1a4101107cd564883
I20260929 18:40:14.548502 23284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:14.553253 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:40:14.560175 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } attempt: 3
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:40:14.565647 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.547872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.0.198.60:42615)
0929 18:40:14.548357 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.548375 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.550402 (+ 2027us) server_negotiation.cc:407] Connection header received
0929 18:40:14.551563 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.551591 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.552061 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.552451 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.553278 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.554727 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.556004 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.556703 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.556861 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.559594 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.559601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.559614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.559646 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.561769 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.562282 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.562290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.562296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.562436 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.562742 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.562745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.562746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.563160 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.563368 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.563676 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.563988 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":343,"thread_start_us":189,"threads_started":1}
I20260929 18:40:14.565956 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.555436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.0.198.62:33333)
0929 18:40:14.555555 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.555559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.555926 (+ 367us) server_negotiation.cc:407] Connection header received
0929 18:40:14.556115 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.556119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.556176 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.556241 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.557159 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.557775 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.558860 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.559127 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.559233 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.561620 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.561623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.561625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.561651 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.563894 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.564389 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.564392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.564394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.564476 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.564825 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.564828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.564830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.565201 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.565306 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.565676 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.565810 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:40:14.566744 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.555359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:33333 (local address 127.0.0.1:43756)
0929 18:40:14.555796 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.555838 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.555960 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.556310 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.556316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.556831 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.557033 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.557041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.557906 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.557909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.558729 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.558741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.559289 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.559300 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.559399 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.559897 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.559909 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.561472 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.564010 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.564019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.564025 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.564287 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.564601 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.564604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.564619 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.564721 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.565329 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.565335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.565467 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.565600 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.566258 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":40,"thread_start_us":182,"threads_started":1}
I20260929 18:40:14.568070 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:14.569932 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.560641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49218)
0929 18:40:14.560782 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.560796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.560881 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.561216 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.561219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.561375 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.561483 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.561491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.562522 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.562526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.563139 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.563149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.563682 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.563685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.563770 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.564211 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.564225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.566175 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.568230 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.568234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.568236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.568528 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.568881 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.568884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.568886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.568981 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.569425 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.569428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.569534 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.569613 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.569750 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 18:40:14.570473 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.560676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.0.198.60:42615)
0929 18:40:14.560971 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.560980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.560993 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:40:14.561026 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.561029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.561070 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.561606 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.561697 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.562416 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.563254 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.563553 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.563972 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.566296 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.566299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.566302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.566326 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.568097 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.568640 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.568643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.568645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.568736 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.569096 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.569100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.569101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.569318 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.570118 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.570193 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.570306 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":110,"threads_started":1}
I20260929 18:40:14.573367 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: Bootstrap starting.
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:14.576129 23209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:14.577019 23209 log.cc:824] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:14.578034 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.568722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.0.198.61:36109)
0929 18:40:14.568846 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.568850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.569070 (+ 220us) server_negotiation.cc:407] Connection header received
0929 18:40:14.569149 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.569152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.569191 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.569255 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.569658 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.570383 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.571415 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.571722 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.571824 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.574530 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.574534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.574537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.574563 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.576236 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.576680 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.576684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.576685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.576770 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.577099 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.577103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.577105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.577342 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.577433 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.577754 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.577872 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:40:14.578293 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.568510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:36109 (local address 127.0.0.1:39100)
0929 18:40:14.569002 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.569013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.569081 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.569280 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.569283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.569431 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.569529 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.569534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.570476 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.570479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.571300 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.571310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.571836 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.571840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.571936 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.572561 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.572575 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.574367 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.576333 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.576337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.576339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.576577 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.576883 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.576885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.576887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.576979 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.577441 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.577444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.577559 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.577670 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.577818 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435}
I20260929 18:40:14.578871 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.568543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.0.198.62:33333)
0929 18:40:14.568705 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.568709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.568721 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:40:14.568754 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.568757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.568800 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.568877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.569623 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.570312 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.571572 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.571816 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.571905 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.574577 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.574580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.574582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.574606 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.576117 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.576892 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.576895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.576897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.576980 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.577722 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.577725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.577727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.577959 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.578185 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.578577 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.578704 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 18:40:14.579186 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:14.579723 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: No bootstrap required, opened a new log
I20260929 18:40:14.583810 23209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
I20260929 18:40:14.584043 23209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e1c28848b64e7eb4d6e2a2c438a7d8, State: Initialized, Role: FOLLOWER
I20260929 18:40:14.584357 23209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:40:14.584802 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.585024 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
I20260929 18:40:14.585778 23209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:14.588133 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.579715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49224)
0929 18:40:14.580019 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.580030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.580104 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.580505 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.580508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.580662 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.580753 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.580759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.581743 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.581746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.582438 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.582447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.582892 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.582896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.582964 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.583430 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.583443 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.584866 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.586597 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.586600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.586602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.586829 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.587095 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.587098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.587099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.587179 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.587646 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.587649 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.587736 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.587837 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.587979 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260929 18:40:14.588340 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.579813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.0.198.60:42615)
0929 18:40:14.579916 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.579920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.580088 (+ 168us) server_negotiation.cc:407] Connection header received
0929 18:40:14.580168 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.580171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.580209 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.580264 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.580865 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.581631 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.582546 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.582777 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.582865 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.584993 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.584996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.584998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.585021 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.586495 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.586922 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.586926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.586928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.587001 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.587273 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.587277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.587279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.587536 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.587655 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.587894 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.588215 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":215,"server-negotiator.queue_time_us":31}
I20260929 18:40:14.590013 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.580414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.198.61:36109)
0929 18:40:14.580503 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.580506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.580634 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:40:14.580729 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.580732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.580772 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.580836 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.581441 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.582066 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.583126 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.583437 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.583537 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.586210 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.586213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.586216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.586241 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.587818 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.588488 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.588491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.588493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.588577 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.588876 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.588879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.588881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.589085 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.589241 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.589744 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.589849 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 18:40:14.591141 23302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:14.591220 23302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:14.591665 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: Bootstrap starting.
I20260929 18:40:14.590408 23141 master_runner.cc:186] Error getting master registration for 127.0.198.62:33333: OK, instance_id { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" instance_seqno: 1790707214065460 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:40:14.593447 23141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:40:14.593858 23284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:14.594894 23284 log.cc:824] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:14.596997 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:14.597203 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: No bootstrap required, opened a new log
I20260929 18:40:14.598385 23108 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43760
I20260929 18:40:14.601023 23284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
I20260929 18:40:14.601274 23284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9c217bac944432aadf1b839992a6dd, State: Initialized, Role: FOLLOWER
I20260929 18:40:14.601612 23284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
I20260929 18:40:14.602113 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.602284 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:14.602978 23284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:40:14.608108 23316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:14.608186 23316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:14.606657 23216 master_runner.cc:186] Error getting master registration for 127.0.198.62:33333: OK, instance_id { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" instance_seqno: 1790707214065460 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:40:14.609920 23216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:40:14.707742 23134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } attempt: 4
Sep 29 18:40:14 dist-test-slave-q5qx krb5kdc[23053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:40:14.718034 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.708343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:42615 (local address 127.0.0.1:49238)
0929 18:40:14.708563 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.708580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:14.708709 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:14.709027 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:14.709030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:14.709246 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:14.709378 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.709386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.710492 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.710496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.711376 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:14.711388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.711957 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.711961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:14.712061 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.712605 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:14.712620 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:14.714267 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:14.716348 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.716351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.716353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.716611 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.716929 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:14.716932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:14.716934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:14.717021 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:14.717511 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:14.717515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:14.717613 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:40:14.717704 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:40:14.717839 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 18:40:14.718524 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.708426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.0.198.60:42615)
0929 18:40:14.708560 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.708565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.708722 (+ 157us) server_negotiation.cc:407] Connection header received
0929 18:40:14.708844 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.708847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.708900 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.708979 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.709518 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.710374 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.711506 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.711850 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.711941 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.714441 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.714445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.714448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.714478 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.716214 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.716719 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.716722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.716724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.716829 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.717114 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.717117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.717119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.717352 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.717464 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.718260 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.718376 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:40:14.721732 23134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222: Bootstrap starting.
I20260929 18:40:14.724828 23134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:14.725766 23134 log.cc:824] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:14.728112 23134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222: No bootstrap required, opened a new log
I20260929 18:40:14.731493 23134 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
I20260929 18:40:14.731745 23134 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bc4d093dc646a5992cf2b0ee8a3222, State: Initialized, Role: FOLLOWER
I20260929 18:40:14.732095 23134 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
I20260929 18:40:14.732690 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.732954 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:14.733551 23134 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:14.733618 23108 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:40:14.733718 23108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:14.734980 23108 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
I20260929 18:40:14.735947 23108 leader_election.cc:288] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [CANDIDATE]: Term 1 election: Requested vote from peers 80e1c28848b64e7eb4d6e2a2c438a7d8 (127.0.198.61:36109), 7d9c217bac944432aadf1b839992a6dd (127.0.198.60:42615)
I20260929 18:40:14.738013 23183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8"
I20260929 18:40:14.738274 23183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:40:14.739269 23330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:14.739352 23330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:14.739658 23183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc4d093dc646a5992cf2b0ee8a3222 in term 1.
I20260929 18:40:14.740142 23257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d9c217bac944432aadf1b839992a6dd"
I20260929 18:40:14.740391 23257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:14.741676 23257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc4d093dc646a5992cf2b0ee8a3222 in term 1.
I20260929 18:40:14.743995 23083 leader_election.cc:302] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9c217bac944432aadf1b839992a6dd, e4bc4d093dc646a5992cf2b0ee8a3222; no voters:
I20260929 18:40:14.749962 23066 master_runner.cc:263] No leader master found from master e4bc4d093dc646a5992cf2b0ee8a3222
I20260929 18:40:14.750082 23066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:40:14.750237 23319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:14.750499 23319 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [term 1 LEADER]: Becoming Leader. State: Replica: e4bc4d093dc646a5992cf2b0ee8a3222, State: Running, Role: LEADER
I20260929 18:40:14.750875 23319 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } }
I20260929 18:40:14.752849 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4bc4d093dc646a5992cf2b0ee8a3222. Latest consensus state: current_term: 1 leader_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.752978 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:14.755622 23338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:14.758517 23338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:14.768553 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.754865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.0.198.60:42615)
0929 18:40:14.754994 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.754999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.758204 (+ 3205us) server_negotiation.cc:407] Connection header received
0929 18:40:14.758382 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.758385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.758432 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.758494 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.761183 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.761893 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.763017 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.763307 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.763375 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.765128 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.765132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.765134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.765159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.766772 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.767203 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.767207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.767209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.767290 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.767606 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.767609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.767611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.767827 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.767919 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.768249 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.768377 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:40:14.774685 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.754200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.0.198.61:36109)
0929 18:40:14.754302 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.754305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.756149 (+ 1844us) server_negotiation.cc:407] Connection header received
0929 18:40:14.756627 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.756630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.756679 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.756721 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.760803 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.761557 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.767727 (+ 6170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.768053 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.768127 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.771019 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.771023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.771026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.771054 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.772654 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.773201 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.773205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.773206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.773303 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.773611 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.773614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.773616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.773825 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.773928 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.774389 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.774516 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:40:14.774685 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.753532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.0.198.62:33333)
0929 18:40:14.753637 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.753642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.755946 (+ 2304us) server_negotiation.cc:407] Connection header received
0929 18:40:14.756390 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.756393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.756438 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.756484 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.762381 (+ 5897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.763104 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.766738 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:14.767069 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:14.767148 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:14.771018 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:14.771023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:14.771026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:14.771054 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:14.772733 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.773201 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.773205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.773206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.773303 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:14.773611 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:14.773614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:14.773616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:14.773825 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:14.773928 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:14.774389 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 18:40:14.774540 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:40:14.781432 23257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [term 1 FOLLOWER]: Refusing update from remote peer e4bc4d093dc646a5992cf2b0ee8a3222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:14.782277 23183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [term 1 FOLLOWER]: Refusing update from remote peer e4bc4d093dc646a5992cf2b0ee8a3222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:14.782663 23319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:14.783111 23319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:14.786829 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4bc4d093dc646a5992cf2b0ee8a3222. Latest consensus state: current_term: 1 leader_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.786988 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:14.787281 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4bc4d093dc646a5992cf2b0ee8a3222. Latest consensus state: current_term: 1 leader_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.787411 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:40:14.787962 23237 tablet.cc:2369] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:40:14.792624 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.792765 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:14.794279 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.794401 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:14.795171 23336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.795316 23336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:14.796216 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bc4d093dc646a5992cf2b0ee8a3222" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 33333 } } peers { permanent_uuid: "80e1c28848b64e7eb4d6e2a2c438a7d8" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 36109 } } peers { permanent_uuid: "7d9c217bac944432aadf1b839992a6dd" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 42615 } } }
I20260929 18:40:14.796306 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:14.796850 23342 mvcc.cc:205] Tried to move back new op lower bound from 7334736751737503744 to 7334736751623270400. Current Snapshot: MvccSnapshot[applied={T|T < 7334736751737503744}]
I20260929 18:40:14.797057 23343 mvcc.cc:205] Tried to move back new op lower bound from 7334736751737503744 to 7334736751623270400. Current Snapshot: MvccSnapshot[applied={T|T < 7334736751737503744}]
I20260929 18:40:14.797484 23344 mvcc.cc:205] Tried to move back new op lower bound from 7334736751737503744 to 7334736751623270400. Current Snapshot: MvccSnapshot[applied={T|T < 7334736751737503744}]
I20260929 18:40:14.799515 23338 catalog_manager.cc:1347] Generated new cluster ID: 3d506ec85cd54604a6fe0037a614890f
I20260929 18:40:14.799645 23338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:14.819252 23338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:14.820503 23338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:14.830065 23338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4bc4d093dc646a5992cf2b0ee8a3222: Generated new TSK 0
I20260929 18:40:14.830931 23338 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:40:14.856653 23138 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.855059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.0.198.62:33333)
0929 18:40:14.855193 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.855197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.855533 (+ 336us) server_negotiation.cc:407] Connection header received
0929 18:40:14.855766 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.855770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.855826 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.855880 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.856508 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43780: BlockingRecv error: recv got EOF from 127.0.0.1:43780 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 18:40:14.856709 23289 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.855060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.0.198.60:42615)
0929 18:40:14.855291 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.855297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.855845 (+ 548us) server_negotiation.cc:407] Connection header received
0929 18:40:14.856008 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.856011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.856072 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.856123 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.856602 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49260: BlockingRecv error: recv got EOF from 127.0.0.1:49260 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
W20260929 18:40:14.856945 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.855530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.198.61:36109)
0929 18:40:14.855644 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.855648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.856009 (+ 361us) server_negotiation.cc:407] Connection header received
0929 18:40:14.856190 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.856194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.856240 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.856289 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.856809 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39136: BlockingRecv error: recv got EOF from 127.0.0.1:39136 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 18:40:14.860190 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.859046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.198.61:36109)
0929 18:40:14.859161 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.859164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.859233 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:40:14.859430 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.859433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.859500 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.859573 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.860004 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39152: BlockingRecv error: recv got EOF from 127.0.0.1:39152 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 18:40:14.864106 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.863327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.0.198.61:36109)
0929 18:40:14.863430 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.863434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.863447 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:40:14.863487 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.863489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.863528 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.863597 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.864006 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39168: BlockingRecv error: recv got EOF from 127.0.0.1:39168 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:40:14.869232 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.868076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.0.198.61:36109)
0929 18:40:14.868182 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.868187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.868338 (+ 151us) server_negotiation.cc:407] Connection header received
0929 18:40:14.868524 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.868526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.868571 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.868648 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.869105 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39174: BlockingRecv error: recv got EOF from 127.0.0.1:39174 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 18:40:14.876533 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.875456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.0.198.61:36109)
0929 18:40:14.875622 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.875626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.875706 (+ 80us) server_negotiation.cc:407] Connection header received
0929 18:40:14.875821 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.875824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.875879 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.875987 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.876417 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39176: BlockingRecv error: recv got EOF from 127.0.0.1:39176 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260929 18:40:14.887974 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.886589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.198.61:36109)
0929 18:40:14.886695 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.886698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.886969 (+ 271us) server_negotiation.cc:407] Connection header received
0929 18:40:14.887194 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.887198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.887262 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.887360 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.887844 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39190: BlockingRecv error: recv got EOF from 127.0.0.1:39190 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260929 18:40:14.907401 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.906113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.0.198.61:36109)
0929 18:40:14.906263 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.906268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.906502 (+ 234us) server_negotiation.cc:407] Connection header received
0929 18:40:14.906634 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.906636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.906684 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.906753 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.907283 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39192: BlockingRecv error: recv got EOF from 127.0.0.1:39192 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 18:40:14.942920 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:14.941747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.0.198.61:36109)
0929 18:40:14.941926 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:14.941931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:14.942045 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:40:14.942207 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:14.942211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:14.942259 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:14.942325 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:14.942789 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39198: BlockingRecv error: recv got EOF from 127.0.0.1:39198 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260929 18:40:15.010733 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:15.009283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.198.61:36109)
0929 18:40:15.009437 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.009442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.009816 (+ 374us) server_negotiation.cc:407] Connection header received
0929 18:40:15.009945 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.009948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.010028 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.010102 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:15.010617 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39214: BlockingRecv error: recv got EOF from 127.0.0.1:39214 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 18:40:15.142421 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:15.141050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.0.198.61:36109)
0929 18:40:15.141247 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.141253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.141475 (+ 222us) server_negotiation.cc:407] Connection header received
0929 18:40:15.141580 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.141584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.141633 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.141703 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:15.142293 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39226: BlockingRecv error: recv got EOF from 127.0.0.1:39226 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 18:40:15.401721 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:15.400715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.0.198.61:36109)
0929 18:40:15.400863 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.400866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.400882 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:40:15.401045 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.401048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.401100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.401173 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:15.401602 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39238: BlockingRecv error: recv got EOF from 127.0.0.1:39238 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:40:15.592511 23302 catalog_manager.cc:1259] Loaded cluster ID: 3d506ec85cd54604a6fe0037a614890f
I20260929 18:40:15.592602 23302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: loading cluster ID for follower catalog manager: success
I20260929 18:40:15.594348 23302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: acquiring CA information for follower catalog manager: success
I20260929 18:40:15.595520 23302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80e1c28848b64e7eb4d6e2a2c438a7d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:40:15.609594 23316 catalog_manager.cc:1259] Loaded cluster ID: 3d506ec85cd54604a6fe0037a614890f
I20260929 18:40:15.609675 23316 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: loading cluster ID for follower catalog manager: success
I20260929 18:40:15.611523 23316 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: acquiring CA information for follower catalog manager: success
I20260929 18:40:15.612699 23316 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d9c217bac944432aadf1b839992a6dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:40:15.920258 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.915934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.0.198.61:36109)
0929 18:40:15.916254 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.916258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.916274 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:40:15.916455 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.916459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.916586 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.916690 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:15.917084 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.917621 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.918617 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.918854 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.918941 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:15.919307 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:15.919785 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:15.919988 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:15.920018 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 18:40:15.920091 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260929 18:40:15.927423 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.923441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.198.60:42615)
0929 18:40:15.923708 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.923712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.923725 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:40:15.923804 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.923806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.923939 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.924054 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:15.924230 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.924783 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.925666 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.925945 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.926092 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:15.926499 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:15.926956 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:15.927066 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:15.927243 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:40:15.927298 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260929 18:40:15.934834 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.930801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.0.198.60:42615)
0929 18:40:15.930981 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.930985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.931311 (+ 326us) server_negotiation.cc:407] Connection header received
0929 18:40:15.931415 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.931419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.931471 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.931536 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:15.931803 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.932553 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.933644 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.933860 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.933944 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:15.934215 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:15.934299 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:15.934358 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:15.934620 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:40:15.934701 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 18:40:15.935261 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.931055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.0.198.61:36109)
0929 18:40:15.931245 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.931249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.931587 (+ 338us) server_negotiation.cc:407] Connection header received
0929 18:40:15.931761 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.931765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.931824 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.931886 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:15.932109 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.932598 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.934009 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.934361 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.934440 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:15.934823 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:15.934889 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:15.934949 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:15.935089 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:40:15.935143 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260929 18:40:15.935541 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.930686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.0.198.62:33333)
0929 18:40:15.930987 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:15.930991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:15.931007 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:40:15.931208 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:15.931211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:15.931355 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:15.931464 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:15.931681 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.932545 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.934009 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:15.934361 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:15.934452 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:15.934791 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:15.935147 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:15.935225 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:15.935328 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:40:15.935386 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":68,"threads_started":1}
I20260929 18:40:15.944802 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 23066
I20260929 18:40:15.950852 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 23141
I20260929 18:40:15.955574 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 23216
2026-09-29T18:40:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3152 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:16 dist-test-slave-q5qx krb5kdc[23432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:16 dist-test-slave-q5qx krb5kdc[23432](info): set up 2 sockets
Sep 29 18:40:16 dist-test-slave-q5qx krb5kdc[23432](info): commencing operation
krb5kdc: starting...
W20260929 18:40:16.918139 792 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/test-user.keytab.
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
Entry for principal kudu/127.0.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab.
I20260929 18:40:17.042714 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:34349
--webserver_interface=127.0.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:34349,127.0.198.61:43853,127.0.198.60:46555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:17.151328 23445 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:40:17.151633 23445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:17.151655 23445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:17.151691 23445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:17.155779 23445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:17.155841 23445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:17.155859 23445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:17.155874 23445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:17.155887 23445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:17.161625 23445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:34349,127.0.198.61:43853,127.0.198.60:46555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.62.keytab
--principal=kudu/127.0.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.62:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:17.162660 23445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:17.163630 23445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:17.171826 23453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:17.171816 23450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:17.171849 23451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:17.172134 23445 server_base.cc:1034] running on GCE node
I20260929 18:40:17.172773 23445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:17.173902 23445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:17.175091 23445 hybrid_clock.cc:648] HybridClock initialized: now 1790707217175069 us; error 39 us; skew 500 ppm
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:17.177837 23445 init.cc:377] Logged in from keytab as kudu/127.0.198.62@KRBTEST.COM (short username kudu)
I20260929 18:40:17.178771 23445 webserver.cc:460] Webserver started at http://127.0.198.62:33171/ using document root <none> and password file <none>
I20260929 18:40:17.179276 23445 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:17.179332 23445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:17.179495 23445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:17.181433 23445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data/instance:
uuid: "384f45cdacad46829533a6c6234bafd3"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "b95ca220bc9bbb965835b182ef17ea2f"
server_key_iv: "eda9d92f8d112ce63c5c8a566baea4f5"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.181926 23445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/wal/instance:
uuid: "384f45cdacad46829533a6c6234bafd3"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "b95ca220bc9bbb965835b182ef17ea2f"
server_key_iv: "eda9d92f8d112ce63c5c8a566baea4f5"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.185680 23445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 18:40:17.188315 23460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:17.189677 23445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:17.189782 23445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/wal
uuid: "384f45cdacad46829533a6c6234bafd3"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "b95ca220bc9bbb965835b182ef17ea2f"
server_key_iv: "eda9d92f8d112ce63c5c8a566baea4f5"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.189875 23445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:17.202116 23445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:17.205170 23445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:17.205324 23445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:17.212365 23445 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.62:34349
I20260929 18:40:17.212641 23512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.62:34349 every 8 connection(s)
I20260929 18:40:17.213340 23445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:17.216205 23513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:17.218851 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 23445
I20260929 18:40:17.218955 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-0/wal/instance
I20260929 18:40:17.219216 792 external_mini_cluster.cc:1428] Setting key 9376880a96b191bc721f9ba8c53dc005
I20260929 18:40:17.220100 23513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.61@KRBTEST.COM" created.
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:17.235615 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.222559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34349 (local address 127.0.0.1:46224)
0929 18:40:17.223111 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.223156 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.224088 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.224780 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.224785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.225213 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.225451 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.225464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.226662 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.226665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.227429 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.227440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.228068 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.228072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.228165 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.228812 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.228828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.230428 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.232440 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.232448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.232460 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.232696 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.233044 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.233047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.233049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.233169 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.233775 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.233781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.233921 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.234181 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.234593 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":103,"spinlock_wait_cycles":3968,"thread_start_us":152,"threads_started":1}
I20260929 18:40:17.235615 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.223580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.198.62:34349)
0929 18:40:17.223907 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.223918 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.223963 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:40:17.224385 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.224389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.224594 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.224758 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.225596 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.226554 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.227545 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.227923 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.228089 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.230602 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.230610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.230627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.230655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.232308 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.232808 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.232814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.232815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.232957 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.233258 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.233261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.233263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.233662 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.233784 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.234265 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.234528 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
I20260929 18:40:17.238124 23513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:17.238829 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.238577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55914)
0929 18:40:17.238691 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.238764 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:40:17.239064 23463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
I20260929 18:40:17.239964 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.239744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55930)
0929 18:40:17.239862 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.239909 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:40:17.240149 23513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:43853: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab.
Entry for principal kudu/127.0.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab.
I20260929 18:40:17.246740 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:43853
--webserver_interface=127.0.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab
--principal=kudu/127.0.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:34349,127.0.198.61:43853,127.0.198.60:46555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:17.289422 23513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } attempt: 1
I20260929 18:40:17.290470 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.290154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55946)
0929 18:40:17.290307 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.290378 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 18:40:17.291257 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.291024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55952)
0929 18:40:17.291139 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.291192 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:40:17.291424 23513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:43853: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
W20260929 18:40:17.355657 23520 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:40:17.355968 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:17.356002 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:17.356055 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:40:17.359634 23513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } attempt: 2
W20260929 18:40:17.360411 23520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:17.360466 23520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:17.360489 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:17.360512 23520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260929 18:40:17.360517 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.360166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55958)
0929 18:40:17.360351 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.360426 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 18:40:17.360539 23520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:17.361205 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.360970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55964)
0929 18:40:17.361104 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.361150 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 18:40:17.361351 23513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.61:43853: Network error: Client connection negotiation failed: client connection to 127.0.198.61:43853: connect: Connection refused (error 111)
I20260929 18:40:17.366180 23520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:34349,127.0.198.61:43853,127.0.198.60:46555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.61.keytab
--principal=kudu/127.0.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.61:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:17.367229 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:17.368523 23520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:17.376816 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:17.376938 23520 server_base.cc:1034] running on GCE node
W20260929 18:40:17.376806 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:17.376806 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:17.377699 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:17.378834 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:17.379987 23520 hybrid_clock.cc:648] HybridClock initialized: now 1790707217379969 us; error 36 us; skew 500 ppm
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:17.383102 23520 init.cc:377] Logged in from keytab as kudu/127.0.198.61@KRBTEST.COM (short username kudu)
I20260929 18:40:17.384024 23520 webserver.cc:460] Webserver started at http://127.0.198.61:45953/ using document root <none> and password file <none>
I20260929 18:40:17.384547 23520 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:17.384606 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:17.384770 23520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:17.386996 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data/instance:
uuid: "008a07d49fa043cc855c9714c7f0e054"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "8a0b5c53f30ab2cbf19a31b3f7f0e647"
server_key_iv: "014dbcfacbdb1fb9303d7672b7247a01"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.387676 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/wal/instance:
uuid: "008a07d49fa043cc855c9714c7f0e054"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "8a0b5c53f30ab2cbf19a31b3f7f0e647"
server_key_iv: "014dbcfacbdb1fb9303d7672b7247a01"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.392815 23520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:40:17.395545 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:17.396909 23520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:40:17.397034 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/wal
uuid: "008a07d49fa043cc855c9714c7f0e054"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "8a0b5c53f30ab2cbf19a31b3f7f0e647"
server_key_iv: "014dbcfacbdb1fb9303d7672b7247a01"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.397147 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:17.408700 23520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:17.411860 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:17.412073 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:17.419028 23520 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.61:43853
I20260929 18:40:17.419333 23587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.61:43853 every 8 connection(s)
I20260929 18:40:17.420022 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/data/info.pb
I20260929 18:40:17.422629 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 23520
I20260929 18:40:17.422751 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-1/wal/instance
I20260929 18:40:17.422979 23588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:17.423091 792 external_mini_cluster.cc:1428] Setting key a0217679d92098e1dbb01b99dddacc6d
I20260929 18:40:17.427634 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.198.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.198.60@KRBTEST.COM" created.
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:17.441922 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.430094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.0.198.62:34349)
0929 18:40:17.430250 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.430254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.431210 (+ 956us) server_negotiation.cc:407] Connection header received
0929 18:40:17.432143 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.432145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.432196 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.432268 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.433377 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.434091 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.435264 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.435609 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.435720 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.438442 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.438445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.438447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.438473 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.439870 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.440396 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.440399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.440400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.440475 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.440800 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.440803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.440805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.440993 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.441087 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.441626 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.441757 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:40:17.442940 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.430482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34349 (local address 127.0.0.1:46238)
0929 18:40:17.431029 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.431089 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.431936 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.432421 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.432428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.432769 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.433239 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.433252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.434261 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.434264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.435138 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.435148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.435779 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.435783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.436033 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.436643 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.436663 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.438267 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.439992 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.439999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.440018 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.440291 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.440568 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.440571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.440573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.440710 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.441097 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.441104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.441380 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.441561 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.441902 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260929 18:40:17.444538 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab.
Entry for principal kudu/127.0.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab.
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
I20260929 18:40:17.450425 792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskal1pt5/build/debug/bin/kudu
/tmp/dist-test-taskal1pt5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:46555
--webserver_interface=127.0.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab
--principal=kudu/127.0.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.198.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.198.62:34349,127.0.198.61:43853,127.0.198.60:46555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:17.452525 23513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } attempt: 3
I20260929 18:40:17.453670 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.445044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55978)
0929 18:40:17.445184 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.445198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.445278 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.445796 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.445800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.445944 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.446099 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.446106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.447236 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.447240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.447853 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.447863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.448513 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.448516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.448604 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.449023 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.449035 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.450257 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.451930 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.451933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.451935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.452113 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.452362 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.452365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.452367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.452435 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.452958 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.452960 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.453264 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.453371 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.453495 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 18:40:17.454206 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.445172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.0.198.61:43853)
0929 18:40:17.445447 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.445456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.445476 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:40:17.445512 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.445515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.445666 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.445779 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.446193 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.447100 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.448011 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.448379 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.448464 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.450386 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.450395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.450407 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.450426 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.451830 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.452207 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.452212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.452214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.452281 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.452509 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.452512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.452514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.452792 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.452923 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.453890 (+ 967us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.454053 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260929 18:40:17.456399 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:17.457188 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.456895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53502)
0929 18:40:17.457033 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.457116 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
W20260929 18:40:17.457772 23536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
I20260929 18:40:17.458943 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.458469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53510)
0929 18:40:17.458826 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.458879 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284}
W20260929 18:40:17.459095 23588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:46555: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
I20260929 18:40:17.461565 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.453006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:55992)
0929 18:40:17.453127 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.453140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.453212 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.453636 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.453639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.453809 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.453942 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.453951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.455015 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.455019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.455556 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.455566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.456043 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.456047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.456150 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.456675 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.456688 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.458009 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.460025 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.460028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.460029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.460267 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.460584 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.460586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.460588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.460669 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.461148 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.461151 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.461239 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.461333 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.461447 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:40:17.461691 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.453080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.198.61:43853)
0929 18:40:17.453399 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.453408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.453419 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:40:17.453447 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.453449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.453489 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.453567 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.454113 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.454873 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.455697 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.455938 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.456023 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.458245 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.458248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.458251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.458272 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.459906 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.460365 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.460369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.460370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.460462 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.460760 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.460763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.460765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.461002 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.461105 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.461411 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.461529 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":143,"threads_started":1}
I20260929 18:40:17.462361 23513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:17.463032 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.462802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53524)
0929 18:40:17.462922 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.462973 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:40:17.463686 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.463436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53536)
0929 18:40:17.463576 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.463641 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 18:40:17.463815 23513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:46555: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
I20260929 18:40:17.495002 23513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } attempt: 1
I20260929 18:40:17.495965 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.495579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53548)
0929 18:40:17.495740 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.495806 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 18:40:17.496824 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.496569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53556)
0929 18:40:17.496710 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.496766 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 18:40:17.497011 23513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:46555: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
I20260929 18:40:17.508347 23588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } attempt: 1
I20260929 18:40:17.509163 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.508879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53564)
0929 18:40:17.509020 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.509086 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:40:17.509832 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.509620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53580)
0929 18:40:17.509723 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.509767 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:40:17.509941 23588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:46555: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
W20260929 18:40:17.565523 23594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:40:17.565809 23594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:17.565830 23594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:17.565874 23594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:17.570003 23594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:17.570055 23594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:17.570072 23594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:17.570086 23594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:17.570099 23594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:17.572181 23513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } attempt: 2
I20260929 18:40:17.573024 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.572688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53582)
0929 18:40:17.572858 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.572930 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 18:40:17.573835 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.573582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53584)
0929 18:40:17.573727 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.573778 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 18:40:17.573998 23513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:46555: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
I20260929 18:40:17.575704 23594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.198.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.198.62:34349,127.0.198.61:43853,127.0.198.60:46555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/krb5kdc/kudu_127.0.198.60.keytab
--principal=kudu/127.0.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.198.60:46555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:40:17.576709 23594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:17.577689 23594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:40:17.578137 23588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } attempt: 2
I20260929 18:40:17.578761 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.578525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53592)
0929 18:40:17.578645 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.578697 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:40:17.579460 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.579233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53608)
0929 18:40:17.579363 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.579410 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 18:40:17.579591 23588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.198.60:46555: Network error: Client connection negotiation failed: client connection to 127.0.198.60:46555: connect: Connection refused (error 111)
W20260929 18:40:17.586548 23604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:17.586541 23601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:17.586545 23602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:17.586740 23594 server_base.cc:1034] running on GCE node
I20260929 18:40:17.587384 23594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:17.588480 23594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:17.589663 23594 hybrid_clock.cc:648] HybridClock initialized: now 1790707217589641 us; error 35 us; skew 500 ppm
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:17.592404 23594 init.cc:377] Logged in from keytab as kudu/127.0.198.60@KRBTEST.COM (short username kudu)
I20260929 18:40:17.593410 23594 webserver.cc:460] Webserver started at http://127.0.198.60:45071/ using document root <none> and password file <none>
I20260929 18:40:17.594000 23594 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:17.594077 23594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:17.594280 23594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:17.596395 23594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data/instance:
uuid: "2c0b54d887634976968bec998f30be65"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "557abd007c1dd3367a0cadaf00df9ed0"
server_key_iv: "100735dd51674ede540aa85edd725a95"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.597002 23594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/wal/instance:
uuid: "2c0b54d887634976968bec998f30be65"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "557abd007c1dd3367a0cadaf00df9ed0"
server_key_iv: "100735dd51674ede540aa85edd725a95"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.600800 23594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:40:17.603324 23611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:17.604563 23594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:40:17.604676 23594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data,/tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/wal
uuid: "2c0b54d887634976968bec998f30be65"
format_stamp: "Formatted at 2026-09-29 18:40:17 on dist-test-slave-q5qx"
server_key: "557abd007c1dd3367a0cadaf00df9ed0"
server_key_iv: "100735dd51674ede540aa85edd725a95"
server_key_version: "encryptionkey@0"
I20260929 18:40:17.604768 23594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:17.628389 23594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:17.631639 23594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:17.631824 23594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:17.638427 23594 rpc_server.cc:307] RPC server started. Bound to: 127.0.198.60:46555
I20260929 18:40:17.638680 23663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.198.60:46555 every 8 connection(s)
I20260929 18:40:17.639364 23594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/data/info.pb
I20260929 18:40:17.642131 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:17.645905 23664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:17.647471 792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskal1pt5/build/debug/bin/kudu as pid 23594
I20260929 18:40:17.647576 792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskal1pt5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706942746699-792-0/minicluster-data/master-2/wal/instance
I20260929 18:40:17.647866 792 external_mini_cluster.cc:1428] Setting key 7f50972a5637f91c502687852af5b4fa
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:40:17.659761 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.647856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.0.198.62:34349)
0929 18:40:17.648012 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.648016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.649115 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 18:40:17.650156 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.650159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.650209 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.650275 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.651087 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.651571 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.652544 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.652754 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.652839 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.655424 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.655428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.655431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.655460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.656941 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.657595 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.657597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.657598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.657668 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.658048 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.658051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.658053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.658268 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.658362 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.659504 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.659623 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:40:17.659969 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.648403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.62:34349 (local address 127.0.0.1:46250)
0929 18:40:17.648956 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.649014 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.649893 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.650342 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.650347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.650694 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.650943 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.650956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.651677 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.651680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.652412 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.652435 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.652887 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.652891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.653119 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.653674 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.653694 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.655271 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.657057 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.657072 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.657082 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.657491 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.657754 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.657757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.657759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.657925 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.658368 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.658373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.658628 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.658776 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.659095 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":43,"thread_start_us":139,"threads_started":1}
I20260929 18:40:17.659970 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.648735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.198.60:46555)
0929 18:40:17.649129 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.649138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.649178 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:40:17.649877 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.649894 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.650198 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.650366 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.651144 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.651950 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.652774 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.653152 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.653238 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.655838 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.655844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.655854 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.655873 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.657497 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.657934 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.657938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.657939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.658045 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.658270 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.658273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.658275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.658651 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.658804 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.659039 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.659188 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":313,"thread_start_us":119,"threads_started":1}
I20260929 18:40:17.661792 23664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.62@KRBTEST.COM
I20260929 18:40:17.670138 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.662236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.61:43853 (local address 127.0.0.1:56002)
0929 18:40:17.662342 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.662354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.662413 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.662775 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.662778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.662917 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.663003 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.663008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.663773 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.663776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.664408 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.664418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.664907 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.664910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.665004 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.665548 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.665558 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.666798 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.668582 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.668585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.668587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.668805 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.669073 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.669075 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.669077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.669161 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.669607 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.669611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.669717 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.669819 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.669943 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:40:17.670190 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.662468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.0.198.61:43853)
0929 18:40:17.662575 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.662578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.662589 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:40:17.662622 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.662624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.662664 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.662739 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.663110 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.663677 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.664524 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.664802 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.664890 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.666919 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.666921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.666923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.666945 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.668472 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.668902 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.668905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.668907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.668986 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.669240 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.669242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.669244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.669485 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.669573 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.669882 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.670029 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:40:17.670717 23588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } attempt: 3
I20260929 18:40:17.670853 23664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:40:17.671540 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.662114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.0.198.62:34349)
0929 18:40:17.662239 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.662243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.662299 (+ 56us) server_negotiation.cc:407] Connection header received
0929 18:40:17.662544 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.662546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.662605 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.662664 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.663289 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.663840 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.664593 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.664791 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.664870 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.667524 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.667527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.667529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.667550 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.669098 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.669629 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.669632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.669634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.669727 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.670163 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.670167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.670169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.670380 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.670589 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.671259 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.671383 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:40:17.673146 23513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } attempt: 3
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.61@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.60@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.198.61@KRBTEST.COM
Sep 29 18:40:17 dist-test-slave-q5qx krb5kdc[23432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.198.62@KRBTEST.COM for kudu/127.0.198.60@KRBTEST.COM
I20260929 18:40:17.681165 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.671158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.0.198.60:46555)
0929 18:40:17.671260 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.671264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.671277 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:40:17.671334 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.671336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.671384 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.672105 (+ 721us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.672192 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.673109 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.674709 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.675139 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.675570 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.677602 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.677606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.677609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.677632 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.679408 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.679870 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.679873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.679874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.679938 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.680243 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.680246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.680247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.680532 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.680626 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.680929 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.681042 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":39}
I20260929 18:40:17.681180 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.671083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53624)
0929 18:40:17.671179 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.671193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.671260 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.671509 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.671513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.671681 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.671772 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.671778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.673780 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.673783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.674591 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.674600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.675264 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.675268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.675354 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.676009 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.676022 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.677475 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.679532 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.679535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.679538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.679765 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.680042 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.680045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.680047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.680132 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.680634 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.680638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.680744 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.680861 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.681010 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:40:17.683542 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.672578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.0.198.61:43853)
0929 18:40:17.672669 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.672673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.672685 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:40:17.672801 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.672804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.672849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.672917 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.673577 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.674284 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.675303 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.675616 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.675703 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.679307 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.679311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.679314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.679356 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.681296 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.681760 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.681763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.681765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.681849 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682186 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.682190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.682191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.682396 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.682580 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.683280 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.683389 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39}
I20260929 18:40:17.683609 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.673503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53634)
0929 18:40:17.673616 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.673630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.673700 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.674281 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.674284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.674466 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.674570 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.674577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.676359 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.676363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.677179 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.677192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.677805 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.677809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.677901 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.678323 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.678337 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.679640 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.682112 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.682117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.682341 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.682651 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.682656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.682740 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.683153 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.683156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.683246 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.683323 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.683466 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:40:17.683720 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.671366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.198.60:46555 (local address 127.0.0.1:53628)
0929 18:40:17.671500 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.671513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:17.671596 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:17.672370 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:17.672374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:17.672536 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:17.672626 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.672633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.674222 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.674224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.675083 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:17.675093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.675728 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.675772 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:17.675867 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.676323 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:17.676337 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:17.678617 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:17.682346 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.682350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.682569 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.682814 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:17.682819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:17.682911 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:17.683296 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:17.683298 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:17.683383 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:40:17.683474 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:40:17.683587 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":26}
I20260929 18:40:17.684219 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.671910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.0.198.60:46555)
0929 18:40:17.672172 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.672181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.672191 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:40:17.672216 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.672219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.672252 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.672331 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.672739 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.674124 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.675194 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.675546 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.675630 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.678749 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.678752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.678755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.678778 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.682254 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682655 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.682658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.682660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.682732 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.683000 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.683003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.683005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.683186 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.683266 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.683969 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.684071 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260929 18:40:17.684831 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.673735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.0.198.60:46555)
0929 18:40:17.674056 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.674066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.674079 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:40:17.674110 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.674113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.674158 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.674677 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.674814 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.675877 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.677327 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.677680 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.677789 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.679778 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.679782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.679784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.679807 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.681891 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682461 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.682464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.682466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.682548 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.682848 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.682851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.682852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.683050 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.684505 (+ 1455us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.684579 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.684682 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":262,"thread_start_us":91,"threads_started":1}
I20260929 18:40:17.684932 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: Bootstrap starting.
I20260929 18:40:17.686897 23664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: Bootstrap starting.
I20260929 18:40:17.687809 23588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:17.687824 23513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3: Bootstrap starting.
I20260929 18:40:17.688678 23588 log.cc:824] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:17.688993 23664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:17.689622 23664 log.cc:824] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:17.690155 23513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:17.690853 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: No bootstrap required, opened a new log
I20260929 18:40:17.690973 23513 log.cc:824] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:17.691354 23664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: No bootstrap required, opened a new log
I20260929 18:40:17.692641 792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:17.695013 23588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.693936 23487 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "384f45cdacad46829533a6c6234bafd3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46258
I20260929 18:40:17.695247 23588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008a07d49fa043cc855c9714c7f0e054, State: Initialized, Role: FOLLOWER
I20260929 18:40:17.695230 23664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.695465 23664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0b54d887634976968bec998f30be65, State: Initialized, Role: FOLLOWER
I20260929 18:40:17.695612 23588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.695727 23513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3: No bootstrap required, opened a new log
I20260929 18:40:17.695771 23664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.696300 23675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.696597 23675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:17.697021 23588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:17.697033 23664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:17.698582 23513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.698820 23513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384f45cdacad46829533a6c6234bafd3, State: Initialized, Role: FOLLOWER
W20260929 18:40:17.702344 23696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:17.702481 23696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 18:40:17.702788 23697 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:17.702884 23697 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:17.701371 23520 master_runner.cc:186] Error getting master registration for 127.0.198.62:34349: OK, instance_id { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" instance_seqno: 1790707217210299 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:40:17.704619 23520 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:40:17.699136 23513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.706494 23513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:40:17.706888 23466 tablet.cc:2369] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:40:17.705225 23594 master_runner.cc:186] Error getting master registration for 127.0.198.62:34349: OK, instance_id { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" instance_seqno: 1790707217210299 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:40:17.708724 23594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:40:17.709048 23487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:40:17.709174 23487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:17.709883 23674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.710057 23674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:17.710512 23487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.711512 23487 leader_election.cc:288] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [CANDIDATE]: Term 1 election: Requested vote from peers 008a07d49fa043cc855c9714c7f0e054 (127.0.198.61:43853), 2c0b54d887634976968bec998f30be65 (127.0.198.60:46555)
I20260929 18:40:17.712248 23562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "384f45cdacad46829533a6c6234bafd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "008a07d49fa043cc855c9714c7f0e054"
I20260929 18:40:17.712499 23562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:17.712759 23637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "384f45cdacad46829533a6c6234bafd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c0b54d887634976968bec998f30be65"
I20260929 18:40:17.713006 23637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:17.713852 23562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384f45cdacad46829533a6c6234bafd3 in term 1.
I20260929 18:40:17.714337 23637 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384f45cdacad46829533a6c6234bafd3 in term 1.
I20260929 18:40:17.714355 23463 leader_election.cc:302] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 008a07d49fa043cc855c9714c7f0e054, 384f45cdacad46829533a6c6234bafd3; no voters:
I20260929 18:40:17.718876 23445 master_runner.cc:263] No leader master found from master 384f45cdacad46829533a6c6234bafd3
I20260929 18:40:17.718858 23700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.718972 23445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:40:17.719003 23700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:17.719091 23700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:17.719170 23700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [term 1 LEADER]: Becoming Leader. State: Replica: 384f45cdacad46829533a6c6234bafd3, State: Running, Role: LEADER
I20260929 18:40:17.719513 23700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } }
I20260929 18:40:17.722048 23700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384f45cdacad46829533a6c6234bafd3. Latest consensus state: current_term: 1 leader_uuid: "384f45cdacad46829533a6c6234bafd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.722232 23700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: This master's current role is: LEADER
W20260929 18:40:17.731220 23711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:17.731326 23711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:17.733067 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.721569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.0.198.61:43853)
0929 18:40:17.721694 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.721699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.723868 (+ 2169us) server_negotiation.cc:407] Connection header received
0929 18:40:17.724062 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.724066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.724118 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.724167 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.725159 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.725996 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.727176 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.727523 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.727597 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.729474 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.729478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.729481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.729509 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.731138 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.731729 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.731733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.731735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.731823 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.732110 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.732114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.732115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.732331 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.732447 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.732787 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.732908 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:40:17.736999 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.723507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.0.198.62:34349)
0929 18:40:17.723632 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.723636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.724485 (+ 849us) server_negotiation.cc:407] Connection header received
0929 18:40:17.724699 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.724702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.724748 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.724795 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.725693 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.726513 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.727691 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.728016 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.728090 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.731597 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.731601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.731604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.731631 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.733194 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.735596 (+ 2402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.735600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.735602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.735704 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.736020 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.736024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.736025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.736241 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.736354 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.736714 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.736841 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:40:17.741626 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.723908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.0.198.60:46555)
0929 18:40:17.724017 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.724022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.730226 (+ 6204us) server_negotiation.cc:407] Connection header received
0929 18:40:17.730646 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.730650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.730703 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.730763 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.733704 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.734470 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.735576 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:17.735963 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:17.736072 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:17.738030 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:17.738035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:17.738037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:17.738068 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:17.739727 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.740240 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.740244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.740246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.740337 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:17.740653 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:17.740656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:17.740658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:17.740873 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:17.740986 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:17.741333 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 18:40:17.741469 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:40:17.742293 23722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:17.743409 23722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:17.748943 23637 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [term 1 FOLLOWER]: Refusing update from remote peer 384f45cdacad46829533a6c6234bafd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:17.750720 23712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:17.755198 23675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384f45cdacad46829533a6c6234bafd3. Latest consensus state: current_term: 1 leader_uuid: "384f45cdacad46829533a6c6234bafd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.755345 23675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:17.758342 23562 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [term 1 FOLLOWER]: Refusing update from remote peer 384f45cdacad46829533a6c6234bafd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:17.758976 23712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:17.763389 23700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "384f45cdacad46829533a6c6234bafd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.763517 23700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:17.763800 23675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "384f45cdacad46829533a6c6234bafd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.763904 23675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:17.764219 23723 mvcc.cc:205] Tried to move back new op lower bound from 7334736763892662272 to 7334736763782594560. Current Snapshot: MvccSnapshot[applied={T|T < 7334736763892662272}]
I20260929 18:40:17.765959 23674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384f45cdacad46829533a6c6234bafd3. Latest consensus state: current_term: 1 leader_uuid: "384f45cdacad46829533a6c6234bafd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.766113 23674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:17.766777 23700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "384f45cdacad46829533a6c6234bafd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.766887 23700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:17.767854 23674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "384f45cdacad46829533a6c6234bafd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384f45cdacad46829533a6c6234bafd3" member_type: VOTER last_known_addr { host: "127.0.198.62" port: 34349 } } peers { permanent_uuid: "008a07d49fa043cc855c9714c7f0e054" member_type: VOTER last_known_addr { host: "127.0.198.61" port: 43853 } } peers { permanent_uuid: "2c0b54d887634976968bec998f30be65" member_type: VOTER last_known_addr { host: "127.0.198.60" port: 46555 } } }
I20260929 18:40:17.767969 23674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:40:17.770083 23724 mvcc.cc:205] Tried to move back new op lower bound from 7334736763892662272 to 7334736763782594560. Current Snapshot: MvccSnapshot[applied={T|T < 7334736763892662272}]
I20260929 18:40:17.773700 23722 catalog_manager.cc:1347] Generated new cluster ID: 764183e321504f15858f2023464f23ed
I20260929 18:40:17.773768 23722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:17.777626 23726 mvcc.cc:205] Tried to move back new op lower bound from 7334736763892662272 to 7334736763782594560. Current Snapshot: MvccSnapshot[applied={T|T < 7334736763892662272}]
I20260929 18:40:17.789927 23722 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:17.791302 23722 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:17.805615 23722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384f45cdacad46829533a6c6234bafd3: Generated new TSK 0
I20260929 18:40:17.808440 23722 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:40:17.844337 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.841039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.198.61:43853)
0929 18:40:17.841187 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.841192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.842966 (+ 1774us) server_negotiation.cc:407] Connection header received
0929 18:40:17.843423 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.843427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.843483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.843537 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.844194 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56030: BlockingRecv error: recv got EOF from 127.0.0.1:56030 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 18:40:17.845880 23517 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.841933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.0.198.62:34349)
0929 18:40:17.842078 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.842083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.845017 (+ 2934us) server_negotiation.cc:407] Connection header received
0929 18:40:17.845187 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.845190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.845241 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.845290 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.845765 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46284: BlockingRecv error: recv got EOF from 127.0.0.1:46284 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 18:40:17.847350 23670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.842589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.0.198.60:46555)
0929 18:40:17.842699 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.842704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.846399 (+ 3695us) server_negotiation.cc:407] Connection header received
0929 18:40:17.846651 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.846654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.846704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.846754 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.847241 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53642: BlockingRecv error: recv got EOF from 127.0.0.1:53642 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:40:17.851886 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.849665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.0.198.61:43853)
0929 18:40:17.849772 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.849776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.850947 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 18:40:17.851169 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.851172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.851248 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.851299 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.851782 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56036: BlockingRecv error: recv got EOF from 127.0.0.1:56036 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:40:17.856828 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.855353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.0.198.61:43853)
0929 18:40:17.855451 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.855455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.855955 (+ 500us) server_negotiation.cc:407] Connection header received
0929 18:40:17.856137 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.856140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.856191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.856242 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.856725 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56042: BlockingRecv error: recv got EOF from 127.0.0.1:56042 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 18:40:17.862888 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.861219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.0.198.61:43853)
0929 18:40:17.861330 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.861334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.861864 (+ 530us) server_negotiation.cc:407] Connection header received
0929 18:40:17.862098 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.862102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.862155 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.862205 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.862739 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56058: BlockingRecv error: recv got EOF from 127.0.0.1:56058 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:40:17.871105 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.869285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.0.198.61:43853)
0929 18:40:17.869434 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.869438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.870072 (+ 634us) server_negotiation.cc:407] Connection header received
0929 18:40:17.870293 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.870297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.870356 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.870413 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.870994 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56060: BlockingRecv error: recv got EOF from 127.0.0.1:56060 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 18:40:17.884461 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.881604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.0.198.61:43853)
0929 18:40:17.881731 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.881735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.882386 (+ 651us) server_negotiation.cc:407] Connection header received
0929 18:40:17.882625 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.882629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.882688 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.882745 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.884333 (+ 1588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56068: BlockingRecv error: recv got EOF from 127.0.0.1:56068 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 18:40:17.908329 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.906419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.0.198.61:43853)
0929 18:40:17.906544 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.906548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.907225 (+ 677us) server_negotiation.cc:407] Connection header received
0929 18:40:17.907428 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.907432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.907488 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.907537 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.908203 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56070: BlockingRecv error: recv got EOF from 127.0.0.1:56070 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:40:17.944603 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:17.943278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.0.198.61:43853)
0929 18:40:17.943435 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:17.943439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:17.943560 (+ 121us) server_negotiation.cc:407] Connection header received
0929 18:40:17.943685 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:17.943687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:17.943734 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:17.943812 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:17.944477 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56072: BlockingRecv error: recv got EOF from 127.0.0.1:56072 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 18:40:18.011960 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:18.010762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.0.198.61:43853)
0929 18:40:18.010913 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.010917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.011008 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:40:18.011194 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.011198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.011249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.011328 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:18.011835 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56076: BlockingRecv error: recv got EOF from 127.0.0.1:56076 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 18:40:18.143476 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:18.142192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.0.198.61:43853)
0929 18:40:18.142362 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.142367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.142516 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:40:18.142661 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.142665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.142715 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.142784 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:18.143345 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56084: BlockingRecv error: recv got EOF from 127.0.0.1:56084 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 18:40:18.402887 23595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:18.401704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.0.198.61:43853)
0929 18:40:18.401906 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.401910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.402015 (+ 105us) server_negotiation.cc:407] Connection header received
0929 18:40:18.402118 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.402121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.402176 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.402254 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:18.402754 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56094: BlockingRecv error: recv got EOF from 127.0.0.1:56094 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 18:40:18.703836 23696 catalog_manager.cc:1259] Loaded cluster ID: 764183e321504f15858f2023464f23ed
I20260929 18:40:18.703915 23696 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: loading cluster ID for follower catalog manager: success
I20260929 18:40:18.703999 23697 catalog_manager.cc:1259] Loaded cluster ID: 764183e321504f15858f2023464f23ed
I20260929 18:40:18.704070 23697 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: loading cluster ID for follower catalog manager: success
I20260929 18:40:18.705547 23696 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: acquiring CA information for follower catalog manager: success
I20260929 18:40:18.705587 23697 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: acquiring CA information for follower catalog manager: success
I20260929 18:40:18.706751 23697 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 008a07d49fa043cc855c9714c7f0e054: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:40:18.706750 23696 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c0b54d887634976968bec998f30be65: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:40:18.921041 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.917065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.0.198.61:43853)
0929 18:40:18.917316 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.917320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.917336 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:40:18.917458 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.917461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.917601 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.917694 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:18.917960 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.918589 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.919450 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.919693 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.919781 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:18.920124 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:18.920614 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:18.920755 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:18.920819 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:40:18.920883 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260929 18:40:18.928681 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.924397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.0.198.60:46555)
0929 18:40:18.924690 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.924694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.924708 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:40:18.924790 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.924792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.924905 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.925111 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:18.925405 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.925934 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.926888 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.927184 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.927274 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:18.927640 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:18.928151 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:18.928256 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:18.928447 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 18:40:18.928519 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":66,"threads_started":1}
I20260929 18:40:18.935997 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.932020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.0.198.61:43853)
0929 18:40:18.932164 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.932168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.932361 (+ 193us) server_negotiation.cc:407] Connection header received
0929 18:40:18.932523 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.932526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.932580 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.932624 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:18.932992 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.933767 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.934809 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.935126 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.935225 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:18.935488 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:18.935577 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:18.935641 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:18.935781 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:40:18.935838 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:40:18.936231 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.932577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.0.198.60:46555)
0929 18:40:18.932687 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.932691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.932701 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:40:18.932739 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.932741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.932778 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.932842 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:18.933243 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.933753 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.934929 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.935258 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.935351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:18.935613 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:18.935686 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:18.935744 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:18.936058 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 18:40:18.936112 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:40:18.936640 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.931979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.198.62:34349)
0929 18:40:18.932212 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:18.932216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:18.932318 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:40:18.932535 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:18.932538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:18.932731 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:18.932821 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:18.933304 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.934086 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.935114 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:18.935439 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:18.935521 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:18.935811 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:18.936194 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:18.936275 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:18.936439 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 18:40:18.936502 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":63,"threads_started":1}
I20260929 18:40:18.940024 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 23445
I20260929 18:40:18.945739 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 23520
I20260929 18:40:18.950834 792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskal1pt5/build/debug/bin/kudu with pid 23594
2026-09-29T18:40:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2995 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6148 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276196 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST